Apparatus, system, and method for physiological sensing in vehicles

ABSTRACT

Methods and apparatus provide physiological movement detection, such as gesture, breathing, cardiac and/or gross motion, such as with sound, radio frequency and/or infrared generation, by electronic devices such as vehicular processing devices. The electronic device in a vehicle may, for example, be any of an audio entertainment system, a vehicle navigation system, and a semi-autonomous or autonomous vehicle operations control system. One or more processors of the device, may detect physiological movement by controlling producing sensing signal(s) in a cabin of a vehicle housing the electronic device. The processor(s) control sensing, with a sensor, reflected signal(s) from the cabin. The processor(s) derive a physiological movement signal with the sensing signal and reflected signal and generate an output based on an evaluation of the derived physiological movement signal. The output may control operations or provide an input to any of the entertainment system, navigation system, and vehicle operations control system.

1 CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication No. 62/609,998, filed Dec. 22, 2017, the entire content ofwhich is incorporated herein by reference.

2 BACKGROUND OF THE TECHNOLOGY 2.1 Field of the Technology

The present technology relates to detecting bio-motion associated withliving subjects with in-vehicle equipment. More particularly, thepresent technology relates to sensing systems for vehicles to detectphysiological characteristics such as physiological movement such asbreathing movement and cardiac movement and/or to detect other lesscyclical body movement of a living subject.

2.2 Description of the Related Art

Monitoring the breathing and body (including limb) movement of a person,for example, prior to or during sleep, can be useful in many ways. Forexample, such monitoring could be useful in detecting sleepiness.Traditionally, the barrier to entry for active radio location or rangingimplementations is that specialized hardware circuitry and antennas arerequired.

Smartphones and other portable and inconspicuous processing orelectronic communication devices have been ubiquitous in daily life,even in developing countries where landlines are not available. Forexample, many vehicles contain audio devices that are capable ofemitting sound with one or more speakers, such as for entertainmentpurposes. Some such systems are configured for electroniccommunications. Many such systems include a microphone for sensing soundso as to serve as part of hands-free system such as for telephone calls,such as when the system is wirelessly coupled to another communicationsdevice (e.g., smart speakers). Such vehicle devices may support voicecommands using virtual assistants that process received verbal commandsand respond with audio output.

It would be desirable to have methods for monitoring bio-motion (i.e.,physiological movement) in an efficient, effective manner in-vehicle.The realization of such a system and method would address a considerabletechnical challenge.

3 BRIEF SUMMARY OF THE TECHNOLOGY

The present technology concerns systems, methods, and apparatus fordetecting movement of a subject, for example, while the subject is awakeor asleep. Based on such movement detection, including for examplebreathing movement, the subject's movements, sleep relatedcharacteristics, respiratory characteristics, cardiac characteristics,sleep state, etc. may be detected. More particularly, an applicationassociated with an electronic device including, for example,processor-enabled equipment, such as a vehicle audio device, in-carentertainment (ICE) device, in-vehicle infotainment (IVI) device orother processing device, such as a smartphone, tablet, smart speaker,guidance (GPS) device, or other hand-held processing device etc. It isclear from the provided examples that the scope of the expression“device”, as well as the meaning of “system” used here, is notnecessarily limited to a single piece of hardware. Any of these termscan encompass a single device, as well as one or more distinct devices.Some or all of these could be integrated in a single piece of equipment,or located separately and remotely from each other. The “device” or“system” is capable of transmitting and/or sensing reflected signals,such as with one or more transmitters and/or sensor(s) (i.e. speaker(s),microphone(s), infrared sensors, radio frequency transmitter/receiveretc.,) being either integrated, and/or externally connectable) to detectphysiological movement.

Some versions of the present technology may include a processor-readablemedium, having stored thereon processor-executable instructions which,when executed by a processor, cause the processor to determinephysiological parameters such as physiological movement of a user.Physiological movement may include any one or more of respirationmovement, cardiac movement, limb movement (e.g., arm or leg), gesturemovement and gross body movement. Apart from the physiological movement,which is a parameter that is derived from at least the detectedreflected signal, physiological parameters may also include one or morecharacteristics that can be further derived from the derivedphysiological movement (e.g., respiratory amplitude, relativerespiratory amplitude, respiratory rate, respiratory rate variability,derived from the respiratory movement signal; relative cardiacamplitude, cardiac amplitude, cardiac rate, cardiac rate variability,derived from the cardiac movement signal, etc.), as well as othercharacteristics (e.g., (a) presence state (present or absent); (b) sleepstate (such as, awake or asleep); (c) sleep stage such as N-REM 1(non-REM light sleep sub-stage 1), N-REM 2 (non-REM light sleepsub-stage 2), N-REM 3 (non-REM deep sleep (also referred to as slow wavesleep (SWS))), REM sleep etc.; or other sleep-related parameters such as(d) fatigue and/or (e) sleepiness; etc.). The processing device may beintegrated with a vehicle (e.g., its cabin) or otherwise be portable oradapted to be inserted into the vehicle cable. The processor-executableinstructions may comprise instructions to control producing, such as viaa speaker coupled to a vehicular audio device or other sensor, a sensingor sound signal within an in-vehicle or cabin vicinity that may includea user. Whilst most of the described examples are applicable to theinside of the vehicle's cabin, similar sensing principles andconsiderations are applicable to the immediate vicinity on the outsideof the vehicle. The processor-executable instructions may compriseinstructions to control sensing, such as via a microphone coupled to thevehicular audio processing device or other sensor, a reflected signal orsound signal reflected from the user in the cabin vicinity of thevehicle. The processor-executable instructions may comprise instructionsto process the sensed sound or reflected signal. Theprocessor-executable instructions may comprise instructions to evaluate,via the microphone coupled to the vehicular audio device, a sensedaudible verbal communication. The processor-executable instructions maycomprise instructions to derive a physiological movement signal with thesensing or sound signal and the reflected signal or reflected soundsignal. The processor-executable instructions may comprise instructionsto generate, such as in response to the sensed audible verbalcommunication, an output based on an evaluation of the derivedphysiological movement signal.

Some versions of the present technology may include a processor-readablemedium having stored thereon processor-executable instructions which,when executed by a processor of an electronic device, cause theprocessor to process data sensed in a cabin vicinity of a vehicle, todetect physiological movement of a user. The processor-executableinstructions may include instructions to control producing a sensingsignal in the cabin vicinity of a vehicle, such as a vehicle housing theelectronic device. The processor-executable instructions may includeinstructions to control sensing, with a sensor, a reflected signal fromthe cabin vicinity of the vehicle. The processor-executable instructionsmay include instructions to derive a physiological movement signal withat least a portion of the sensed reflected signal and a signalrepresentative of a portion of the sensing signal. Theprocessor-executable instructions may include instructions to generatean output based on an evaluation of at least a portion of the derivedphysiological movement signal.

In some versions, the sensing signal may be any one or more of a radiofrequency sensing signal generated by a radio frequency transmittercoupled with the electronic device, an acoustic sensing signal generatedby a speaker coupled with the electronic device, and an infrared sensingsignal generated by an infrared emitter coupled with the electronicdevice. The signal representative of the portion of the sensing signalmay include an internally generated oscillator signal or a direct pathmeasured sound signal. The instructions to derive the physiologicalmovement signal derive may be configured to derive the physiologicalmovement signal (a) with the sensing signal and the reflected signal; or(b) with the reflected signal and an associated signal that isassociated with the sensing signal, optionally wherein the associatedsignal is an internally generated oscillator signal or a direct pathmeasured signal.

The instructions to derive the physiological movement signal may beconfigured to multiply an oscillator signal with the portion of thesensed reflected signal. The derived physiological movement signal mayinclude one or more of a respiratory motion, gross motion or a cardiacmotion of a user within the cabin vicinity. The evaluation of thederived physiological movement signal may include determining any one ormore of breathing rate, amplitude of breathing, relative amplitude ofbreathing, cardiac rate, cardiac amplitude and relative cardiacamplitude. The processor-executable instructions may includeinstructions to sense vehicle characteristics based on a signal from oneor more vehicle sensors and generate the output based on the sensedvehicle characteristics. The processor-executable instructions mayinclude instructions to sense vehicle characteristics based on a signalfrom one or more vehicle sensors, and to adjust at least a portion ofthe produced sensing signal based on the sensed vehicle characteristics.

In some versions, the sensed vehicle characteristics may include any oneor more of vehicle speed, door opening state, window opening state,engine revolutions, vehicle location, seat occupancy, seatbelt fasteningstate, seat position, steering wheel grip status, steering wheel angle,air conditioning system status, fan setting, brake setting, gas pedalsetting, cabin light, cabin noise, and/or cabin temperature. Theprocessor-readable medium may include further comprisingprocessor-executable instructions to evaluate, via a microphone coupledto the electronic device, a sensed audible verbal communication; andwherein the generated output based on an evaluation of the derivedphysiological movement signal is further based on the sensed audibleverbal communication. The electronic device may comprise an audioentertainment system, wherein the sensing signal may be combined with anaudio entertainment content signal, and wherein the combined sensingsignal and audio entertainment content signal may be produced by one ormore speakers of the audio entertainment system. At least a portion ofthe produced sensing signal may be a sound signal in a substantiallyinaudible sound range. The sound signal may be a low frequencyultrasonic acoustic signal. The sound signal may be a dual tonefrequency modulated continuous wave signal. The dual tone frequencymodulated continuous wave signal may comprise a first sawtooth frequencychange at least partially overlapped with a second sawtooth frequencychange in a repeated waveform.

The audio entertainment system may include a plurality of speakers andwherein the electronic device may be configured to derive differentphysiological movement signals, each derived physiological movementsignal being associated with a different speaker of the plurality ofspeakers. The instructions to control producing a sensing signal mayproduce sensing signals in different sensing frequency ranges for eachdifferent speaker of the plurality of speakers. The instructions tocontrol sensing the reflected signal from the cabin vicinity of thevehicle, may control sensing of reflected signals by using a pluralityof microphones. The medium may further include processor-executableinstructions to control the electronic device generate, with a speaker,a sound presentation to either discourage or promote sleep by the userin the cabin vicinity. The sound presentation may include a breathingentrainment exercise.

The electronic device may comprise a vehicle navigation system. Theprocessor executable-instructions of the electronic device may beconfigured to, based on the output from the evaluation of the derivedphysiological movement signal, set a parameter for a navigation routeprovided with the vehicle navigation system. The electronic device maycomprise a semi-autonomous or autonomous vehicle operations controlsystem. The processor executable-instructions of the electronic devicemay be configured to, based on the output from the evaluation of thederived physiological movement signal, control any one or more of:movement of the vehicle, adjustment of a light condition of the cabinvicinity, adjustment of electrochromic glass transparency, movement of aseat of the cabin vicinity, adjustment of a braking parameter,adjustment of an acceleration parameter, adjustment of a suspensionsetting, adjustment of window coverage, adjustment of an acousticbarrier, immobilization of the vehicle, engagement of vehicleventilation and/or engagement of vehicle cabin cooling/heating system.The evaluation of the derived physiological movement signal may includedetection of any one or more of sleepiness, fatigue state, a sleep stageand a time in a sleep stage and/or a calculation of sleep score. Theevaluation of the portion of the derived physiological movement signalmay include detection of any one any of a sleep score, a sleep stage anda time in a sleep stage. The evaluation of the portion of the derivedphysiological movement signal by the electronic device may include asleep service.

In some versions, the evaluation of the portion of the derivedphysiological movement signal by the electronic device may include ahealth screening service. Optionally, the health screening service mayinclude any one or more of detection of respiratory health, detection ofsleep disordered breathing, and detection of cardiac health. Theevaluation of the portion of the derived physiological movement signalcomprises detection of a gesture. The evaluation of the portion of thederived physiological movement signal by the electronic device mayinclude detection of one or more of respiratory health relatedparameters, sleep disordered breathing related parameters, and/orcardiac health related parameters.

In some versions, the processor-readable medium may includeprocessor-executable instructions to generate an ultra-wide band (UWB)sound sensing signal as audible white noise. The processor-readablemedium may include processor executable instructions to detect usermotion with the UWB sound signal. The medium may further includeprocessor-executable instructions to generate, in a setup process,probing signals to map distances within the cabin vicinity. The mediummay further include processor-executable instructions to detect presenceor absence of a user in the cabin vicinity based on the portion of thederived physiological movement signal. The medium may further includeprocessor-executable instructions to conduct biometric recognition of auser in the cabin vicinity based on the portion of the derivedphysiological movement signal. Optionally, the output may include agenerated alert. The medium may further include processor-executableinstructions to control enabling and disabling a vehicle operationscontrol system of the vehicle based the biometric recognition. Themedium may include processor-executable instructions to conductbiometric recognition of a user in the cabin vicinity based on theportion of the derived physiological movement signal. The output may bebased on the biometric recognition and may include at least one of: (a)generating an alert; and (b) controlling enabling and disabling avehicle operations control system of the vehicle. The medium may furtherinclude comprising processor-executable instructions to filter a soundsignal sensed by a microphone coupled to the electronic device. Thefilter may be configured to mitigate or attenuate vehicular sounds. Thevehicular sounds may include one or more of: motor noise, wind noise, acar horn, a door closing, and infotainment sounds.

In some versions, the evaluation of the portion of the derivedphysiological movement signal by the electronic device may includeclassification of the derived physiological movement signal, wherein theclassification evaluates features determined from the portion of thederived physiological movement signal by a deep belief network. Theevaluation of the derived physiological movement signal by theelectronic device may include determination of a child remaining alonein the cabin vicinity, and wherein the output comprises a generatedwarning. The output may further include a control signal to activate avehicle operations control system based on the determination of a childremaining alone in the cabin vicinity. Optionally, the vehicleoperations control system may include a vehicle environment controlsystem and the control signal may initiate a ventilation and/ortemperature condition of the cabin vicinity provided by the vehicleenvironment control system. The evaluation of the portion of the derivedphysiological movement signal by the electronic device may include adetermination of a child remaining alone in the cabin vicinity, whereinthe output may include: (a) a generated warning, or (b) the vehicleoperations control system initiating a ventilation and/or temperaturecondition of the cabin vicinity.

In some versions, the medium may further include processor-executableinstructions to record data based on the derived physiological movementsignal in a blockchain data system. The medium may further includeprocessor-executable instructions to generate the output as aninteractive language process through a chatbot program. The electronicdevice may comprise a hand-held processing device. The electronic devicemay include one or more integrated components of a vehicle or avehicular processing device. In some versions, one or both of (a) thesensor and (b) a component configured to produce the sensing signal, maybe an integrated component(s) of a vehicle.

Some versions of the present technology may include a server with accessto the processor-readable medium as described herein. The server may beconfigured to receive requests for downloading the processor-executableinstructions of the processor-readable medium to an electronic device ora vehicular processing device over a network.

Some versions of the present technology may include an electronicdevice. The electronic device may comprise one or more processorsarranged to be coupled to a sensor operating in a cabin vicinity of avehicle; and (a) any processor-readable medium as described herein, or(b) a processor-readable medium configured to access theprocessor-executable instructions of any server described herein. Thesensor may comprise at least one of (a) a speaker and microphone, (b) aninfrared emitter and detector, or (c) a radio frequency transceiver. Theelectronic device may include any one of more of an audio entertainmentsystem, a vehicle navigation system, and a semi-autonomous or autonomousvehicle operations control system. The electronic device may include oneor more integrated components of a vehicle or a vehicular processingdevice. The electronic device may further comprise the vehicle. Theelectronic device may include at least one portable component. Theportable component may include a smart phone, a smart watch or smartjewelry.

Some versions of the present technology may include a method of a serverhaving access to any processor-readable medium described herein, or tothe electronic device described herein. The method may includereceiving, at the server, a request for downloading theprocessor-executable instructions of the processor-readable medium tothe electronic device over a network; and transmitting theprocessor-executable instructions to the electronic device in responseto the request.

Some versions of the present technology may include a method of aprocessor of an electronic device. The method may include accessing,with the processor, any of the processor-readable medium(s) describedherein. The method may include executing, in the processor, theprocessor-executable instructions of the processor-readable medium.

Some versions of the present technology may include a method of one ormore processors of an electronic device to detect physiological movementof a user in a cabin vicinity of a vehicle. The method may includecontrolling producing a sensing signal in the cabin vicinity of thevehicle. The method may include controlling sensing, with a sensor, areflected signal from the cabin vicinity of the vehicle. The method mayinclude deriving a physiological movement signal with at least a portionof the sensed reflected signal and a signal representative of a portionof the sensing signal. The method may include generating an output basedon an evaluation of at least a portion of the derived physiologicalmovement signal.

In some versions of the method the sensing signal may be any one or moreof a radio frequency sensing signal generated by a radio frequencytransmitter coupled with the electronic device, an acoustic sensingsignal generated by a speaker coupled with the electronic device, and aninfrared sensing signal generated by an infrared emitter coupled withthe electronic device. The signal representative of the portion of thesensing signal may include an internally generated oscillator signal ora direct path measured signal. The method may include deriving thephysiological movement signal (a) with the sensing signal and the sensedreflected signal; or (b) with the sensed reflected signal and anassociated signal that is associated with the sensing signal, optionallywherein the associated signal is an internally generated oscillatorsignal or a direct path measured signal. The method may include derivingthe physiological movement signal comprises multiplying an oscillatorsignal with the portion of the sensed reflected sound signal. Thederived physiological movement signal may include one or more of arespiratory motion, a cardiac motion, or gross motion, of a user withinthe cabin vicinity. The evaluation of the portion of the derivedphysiological movement signal may include determining any one or more ofbreathing rate, relative amplitude of breathing, amplitude of breathing,cardiac rate, relative cardiac amplitude, and cardiac amplitude. Themethod may include sensing vehicle characteristics based on a signalfrom one or more vehicle sensors and generating the output based on thesensed vehicle characteristics. The method may include sensing vehiclecharacteristics based on a signal from one or more vehicle sensors, andto adjust at least a portion of the produced sensing signal based on thesensed vehicle characteristics. The sensed vehicle characteristics mayinclude any one or more of vehicle speed, door opening state, windowopening state, engine revolutions, vehicle location, seat occupancy,seatbelt fastening state, seat position, steering wheel grip status,steering wheel angle, air conditioning system status, fan setting, brakesetting, gas pedal setting, cabin light, cabin noise, and/or cabintemperature.

The method may include further evaluating, via a microphone coupled tothe electronic device, a sensed audible verbal communication; andwherein the generated output based on an evaluation of the portion ofthe derived physiological movement signal is further based on the sensedaudible verbal communication. The electronic device may include an audioentertainment system and the method may include combining the sensingsignal with an audio entertainment content signal, and producing thecombined sensing signal and audio entertainment content signal by one ormore speakers of the audio entertainment system. At least a portion ofthe produced sensing signal may be a sound signal in a substantiallyinaudible sound range. The sound signal may be a low frequencyultrasonic acoustic signal. The sound signal may be a dual tonefrequency modulated continuous wave signal. The dual tone frequencymodulated continuous wave signal comprises a first sawtooth frequencychange at least partially overlapped with a second sawtooth frequencychange in a repeated waveform. The electronic device may include anaudio entertainment system that may include a plurality of speakers andwherein the electronic device derives different physiological movementsignals, each derived physiological movement signal being associatedwith a different speaker of the plurality of speakers.

In some versions, the controlling producing a sensing signal producessensing signals in different sensing frequency ranges for each differentspeaker of the plurality of speakers. The controlling sensing thereflected signal from the cabin vicinity of the vehicle may includecontrolling sensing of reflected signals by using a plurality ofmicrophones. The method may include controlling the electronic device togenerate, with a speaker, a sound presentation to either discourage orpromote sleep by the user in the cabin vicinity. The sound presentationmay include a breathing entrainment exercise. The electronic device mayinclude a vehicle navigation system. The electronic device, based on theoutput from the evaluation of the portion of the derived physiologicalmovement signal, may set a parameter for a navigation route providedwith the vehicle navigation system.

The electronic device may include a semi-autonomous or autonomousvehicle operations control system. The electronic device may control,based on the output from the evaluation of the derived physiologicalmovement signal, any one or more of: movement of the vehicle, adjustmentof a light condition of the cabin vicinity, adjustment of electrochromicglass transparency, movement of a seat of the cabin vicinity, adjustmentof a braking parameter, adjustment of an acceleration parameter,adjustment of a suspension setting, adjustment of window coverage,adjustment of an acoustic barrier, immobilization of the vehicle,engagement of vehicle ventilation and/or engagement of vehicle cabincooling/heating system. The evaluation of the portion of the derivedphysiological movement signal may include detecting any one or more ofsleepiness, fatigue state, a sleep stage and a time in a sleep stageand/or a calculation of sleep score. The evaluation of the portion ofthe derived physiological movement signal may include detecting any oneof a sleep score, a sleep stage and a time in a sleep stage. Theevaluation of the portion of the derived physiological movement signalby the electronic device may include a sleep service. The evaluation ofthe portion of the derived physiological movement signal by theelectronic device may include a health screening service, and optionallywherein the health screening service detects any one or more ofrespiratory health, sleep disordered breathing, and cardiac health. Theevaluation of the portion of the derived physiological movement signalmay detect a gesture. The evaluation of the portion of the derivedphysiological movement signal by the electronic device may includedetection of any one or more of respiratory health related parameters,sleep disordered breathing related parameters, and cardiac healthrelated parameters.

The produced sensing signal may include ultra-wide band (UWB) soundsensing signal generated as audible white noise. The method may furtherinclude generating an ultra-wide band (UWB) sound sensing signal asaudible white noise, and detecting user motion with the UWB soundsignal. The method may include generating, in a setup process, probingsignals to map distances within the cabin vicinity. The method mayfurther include detecting presence and absence of a user in the cabinvicinity based on the derived physiological movement signal. The methodmay further include conducting biometric recognition of a user in thecabin vicinity based on the derived physiological movement signal.Optionally, the output may include a generated alert. The method mayfurther include controlling enabling and disabling a vehicle operationscontrol system of the vehicle based the biometric recognition. Theoutput may be based on the biometric recognition and may include atleast one of (i.e., or both): (a) generating an alert and (b)controlling enabling and disabling a vehicle operations control systemof the vehicle. The method may further include filtering a sound signalsensed by a microphone coupled to the electronic device, the filteringmay be configured to mitigate or attenuate vehicular sounds. Optionally,the vehicular sounds may include one or more of: motor noise, windnoise, a car horn, a door closing, and infotainment sounds.

The evaluation of the derived physiological movement signal by theelectronic device may include classifying the derived physiologicalmovement signal, wherein the classifying evaluates features determinedfrom the derived physiological movement signal by a deep belief network.The evaluation of the derived physiological movement signal by theelectronic device may include determining presence of a child remainingalone in the cabin vicinity, and wherein the output comprises agenerated warning. The output may include a control signal to activate avehicle operations control system based on the determining presence of achild remaining alone in the cabin vicinity. The evaluation of theportion of the derived physiological movement signal by the electronicdevice may include determining a presence of a child remaining alone inthe cabin vicinity, and the output may include: (a) a generated warning,or (b) the vehicle operations control system initiating a ventilationand/or temperature condition of the cabin vicinity provided by thevehicle environment control system. Optionally, the vehicle operationscontrol system may include a vehicle environment control system and thecontrol signal may initiate a ventilation and/or temperature conditionof the cabin vicinity provided by the vehicle environment controlsystem. The method may further include recording data based on thederived physiological movement signal in a blockchain data system. Themethod may further include generating the output as an interactivelanguage process through a chatbot program. The electronic device mayinclude, or be, a hand-held processing device. The electronic device mayinclude one or more integrated components of a vehicle or a vehicularprocessing device. One or both of (a) the sensor and (b) a componentconfigured to produce the sensing signal, may be an integratedcomponent(s) of a vehicle.

The methods, systems, devices and apparatus described herein can provideimproved functioning in a processor, such as of a processor of anin-vehicle audio and/or processing device, a general or specific purposecomputer, portable computer processing device (e.g., mobile phone,tablet computer, smart speaker, smart television etc.), respiratorymonitor and/or other processing apparatus utilizing a motion sensor suchas a microphone and speaker. Moreover, the described methods, systems,devices and apparatus can provide improvements in the technologicalfield of automated vehicle audio apparatus.

Of course, portions of the aspects may form sub-aspects of the presenttechnology. Also, various ones of the sub-aspects and/or aspects may becombined in various manners and also constitute additional aspects orsub-aspects of the present technology.

Other features of the technology will be apparent from consideration ofthe information contained in the following detailed description,abstract, drawings and claims.

4 BRIEF DESCRIPTION OF THE DRAWINGS

The present technology is illustrated by way of example, and not by wayof limitation, in the figures of the accompanying drawings, in whichlike reference numerals refer to similar elements including:

FIG. 1 illustrates an example in-vehicle motion sensing device, such asone using low frequency ultrasonic biomotion sensing, with signalgeneration and processing techniques described herein.

FIG. 2 illustrates an example processing device for receiving audioinformation from an in-vehicle vicinity of the device and a schematicillustration of example processes of the device.

FIG. 3 is schematic illustration of a processing device, such as a smartaudio device, as it may be configured in accordance with some forms ofthe present technology.

FIG. 4A shows frequency characteristics of a single tone chirp such asfor frequency modulated continuous wave sensing (FMCW).

FIG. 4B shows frequency characteristics of a dual tone chirp such as forfrequency modulated continuous wave sensing (FMCW).

FIG. 5 illustrates example demodulation for a dual tone FMCW that may beimplemented for a sensing system in a vehicular processing device of thepresent technology.

FIG. 6 illustrates example operations of voice enabled audio device orvehicular processing device such as one using low frequency ultrasonicbiomotion sensing with signal generation and processing techniquesdescribed herein.

FIG. 7 illustrates example audio processing modules or blocks such asfor the processing of the vehicular processing device(s) describedherein.

FIG. 8 illustrates example processing flow for in-vehicle motion sensingsuch as for generating various output (e.g., sleep data,fatigue/alertness data, medical data, a notification data) based onmotion characteristics, such as for the sensing device illustrated inFIG. 1.

FIGS. 8A, 8B and 8C illustrate various signal characteristics of atriangular single tone such as for a FMCW system of the presenttechnology.

FIG. 9 illustrates example processing flow for an in-vehicle acousticsensing apparatus, such as a vehicular processing device enabled withSONAR physiological sensing apparatus.

FIGS. 9A, 9B and 9C illustrate various signal characteristics of atriangular dual tone such as for a FMCW system of the presenttechnology.

FIG. 10 illustrates example processing flow for a process for in-vehiclesensing of physiological parameters to implement an in-vehicle healthassessment processing system and/or an in-vehicle sleep service system.

FIG. 11 illustrates example processing flow for a process with sensingof physiological parameters to implement an in-vehicle nap/sleep servicesystem.

5 DETAILED DESCRIPTION OF EXAMPLES OF THE TECHNOLOGY

Before the present technology is described in further detail, it is tobe understood that the technology is not limited to the particularexamples described herein, which may vary. It is also to be understoodthat the terminology used in this disclosure is for the purpose ofdescribing particular examples discussed and is not intended to belimiting.

The following description is provided in relation to various forms ofthe present technology that may share common characteristics orfeatures. It is to be understood that one or more features of any oneexemplary form may be combinable with one or more features of anotherform. In addition, any single feature or combination of features in anyof form described herein may constitute a further exemplary form.

5.1 Screening, Monitoring, and Detection with in-Vehicle Equipment

The present technology concerns physiological sensing systems, methods,and apparatus for detecting movement of a subject, including, forexample, gross body movement, breathing movement and/or cardiac relatedchest movement, such as while the subject is in a vehicular environment.More particularly, the technology concerns processing applicationsassociated with in-vehicle motion sensing device(s), such as for anautomobile entertainment system. In some versions the sensing device maybe a smartphone, a guidance system, an in-car audio system, a tablet, amobile device, a mobile phone, a smart television, a laptop computer,etc. that uses the device sensors, such as a speaker and microphone, todetect such in-vehicle motion. Such as electronic device system mayinclude a portable component, such as a smart phone, smart watch, and/orsmart jewelry.

A particularly minimalist or non-obtrusive version of an example systemsuitable for implementing the present technology is now described withreference to FIGS. 1 to 3. The in-vehicle audio device may beimplemented as a processing device 100 having one or more processors,such as a microcontroller, that may be a smart speaker that isconfigured with an application 200 for detecting movement of subject110. It may be placed within a vehicle 111 or otherwise integrated withcomponents of a cabin (e.g., audio system, dashboard, doors, seats etc.)of a vehicle 111 near subject 110. Optionally, processing device 100 maybe, for example, a smartphone, tablet computer, laptop computer, smartspeaker, smart television or other electronic device. The processor(s)of the processing device 100 may be configured to, among other things,execute the functions of application 200, including causing a sensingsignal 112, such as an acoustic or audio signal, to be generated andtransmitted, typically through the air in a generally restrictedvicinity of a vehicle. The processing device may receive a reflection114 of the transmitted signal by sensing it with, for example, atransducer such as a microphone. The processing device may process thesensed signal, such as by demodulation with the transmitted signal, todetermine body movement such gross body movement, limb movement, cardiacmovement and respiration movement. Processing device 100 will maycomprise, among other components, a speaker and a microphone. Thespeaker may be implemented to transmit the generated audio signal andthe microphone to receive the reflected signal. The generated audiosignal for sensing and processing may be implemented with any of thetechniques described in International Patent ApplicationPCT/EP2017/073613 filed on Sep. 19, 2017, the entire disclosure of whichis incorporated herein by reference.

While the sensing apparatus are generally described herein in relationto acoustic sensing (e.g., low frequency ultrasonic sensing), it isunderstood that the methods and devices may be implemented using othersensing techniques. For example, as an alternative, the processingdevice may be implemented with a radio frequency transceiver of an RFsensor to serve as sensing apparatus, such that the generated signal andreflected signal are RF signals. Such a RF sensing device, which may beintegrated with or coupled to the processing device, may implementedwith any of the techniques and sensor components described inInternational Patent Application No. PCT/US2013/051250, entitled “RangeGated Radio Frequency Physiology Sensor” and filed on Jul. 19, 2013;International Patent Application No. PCT/EP2017/070773, entitled“Digital Radio Frequency Motion Detection Sensor” and filed on Aug. 16,2017; and International Patent Application No. PCT/EP2016/069413,entitled “Digital Range Gated Radio Frequency Sensor” and filed on Aug.16, 2017. Similarly, in alternative versions, such sensing apparatus forthe transmission of a sensing signal and sensing of its reflection maybe implemented with an infrared radiation generator and an infraredradiation detector (e.g., an IR emitter and IR detector). The processingof such signals for motion detection and characterization as describedherein may be similarly implemented.

Using a combination of two or more of these different sensingtechniques, can enhance the sensing outcome by combining the advantagesof the respective techniques. For instance, the discussed acousticsensing technique is quite acceptable in the noisy environment of, forexample, city driving or higher engine revolutions/vehicle speeds.However, a user with very sensitive hearing may find the use of thistechnique to be problematic, such as during country driving or at lowerengine revolutions/vehicle speeds, when the noise is much lower and thesensing signal is easier to hear. Similarly, whilst an IR sensingprovides a good S/N signal during night time, its use may be problematicin the light (and heat) of day. An IR sensing may be used in this caseat night, complemented by the use of the acoustic sensing during theday.

Optionally, the sensing methodologies of the processing device may beimplemented with sensing apparatus in or by other types of devices suchas a travel/portable sleeping therapy device (e.g., a travel/portablerespiratory therapy device such as a continuous positive airway pressure(e.g., “CPAP”) device or high flow therapy device) (not shown) where thetherapy device serves as the processing device 100 or works inconjunction with a separate processing device 100. Examples of suchdevices, including a pressure device or blower (e.g., a motor andimpeller in a volute), one or more sensors and a central controller ofthe pressure device or blower, may be considered in reference to thedevices described in International Patent Publication No. WO/2015/061848(Appl. No. PCT/AU2014/050315) filed on Oct. 28, 2014, and InternationalPatent Publication No. WO/2016/145483 (Appl. No. PCT/AU2016/050117)filed on Mar. 14, 2016, the entire disclosures of which are incorporatedherein by reference. Such a respiratory therapy device may include anoptional humidifier 4000 and provide therapy to a patient interface viaa patient circuit (e.g., a conduit). In some cases, the respiratorytherapy device might have a separate sensor, such as a microphone, forsensing internal sound-related conditions within and through the patientcircuit, as opposed to serving to sense the externally sound relatedacoustic conditions of the processes described throughout thisapplication.

Processing device 100 may be adapted to provide an efficient andeffective method of monitoring a subject's breathing and/or othermovement related characteristics. When used within a vehicle, such asduring sleep, the processing device 100 and its associated methods canbe used to detect, for example, the user's breathing and identify sleepstages, sleep states, transitions between states, breathing and/or otherrespiratory characteristics. When used during wake, the processingdevice 100 and its associated methods can be used to detect movementsuch as presence or absence of a person or subject breathing(inspiration, expiration, pause, and derived rate), sleepiness and/orfatigue, a ballistocardiogram waveform and/or subsequent derived heartrate, etc. Such movement or movement characteristics may be used forcontrolling various functions as described herein in more detail.

Processing device 100 may include integrated chips, a memory and/orother control instruction, data or information storage medium. Forexample, programmed instructions encompassing the assessment/signalprocessing methodologies described herein may be coded on integratedchips in the memory of the device or apparatus to form an applicationspecific integrated chip (ASIC). Such instructions may also oralternatively be loaded as software or firmware using an appropriatedata storage medium. Optionally, such processing instructions may bedownloaded such as from a server over a network (e.g. an internet) tothe processing device such that when the instructions are executed, theprocessing device serves as a screening or monitoring device.

Accordingly, processing device 100 may include a number of components asillustrated by FIG. 3. The processing device 100 may include, amongother components, sensing apparatus 309, such as a microphone(s) orsound sensor 302 and a speaker 310 for acoustic sensing, a processor(s)304, an optional display interface 306, an optional user control/inputinterface 308, and a memory/data storage 312, such as with theprocessing instructions of the processing methodologies/modulesdescribed herein. In some cases, the microphone and/or speaker may serveas the user interface with the processor(s) of the device, such as tocontrol operations of the processing device, for example, when theprocessing device responds, such as via the speaker, to audio and/orverbal commands sensed by the microphone. In this regard, the processingdevice 100 may serve as a voice assistant such as using natural languageprocessing.

One or more of the components of processing device 100 may be integralwith or operably coupled with processing device 100. For example,sensing apparatus 309 (e.g., microphone(s) or sound sensor 302 andspeaker 310) may be integral with processing device 100 or coupled withprocessing device 100 such as through a wired or wireless link (e.g.,Bluetooth, Wi-Fi etc.). Thus, the processing device 100 may include adata communications interface 314.

Memory/data storage 312 may comprise a plurality of processor controlinstructions for controlling processors 304. For example, memory/datastorage 312 may comprise processor control instructions for causingapplication 200 to be performed by the processing instructions of theprocessing methodologies/modules described herein.

Examples of the present technology may be configured to use one or morealgorithms or processes, which may be embodied by application(s) 200, todetect motion, breathing, and optionally sleep characteristics while auser is asleep using the processing device 100. For example, application200 may be characterized by several sub-processes or modules. As shownin FIG. 2, application 200 may include a sensing signal generation andtransmission sub-process 202, a motion and bio-physical characteristicdetection sub-process 204, a motion characterization sub-process 206such as for subject absence/presence detection, biometricidentification, fatigue, sleepiness, sleep characterization, respiratoryor cardiac related characterizations etc., and a results outputsub-process 208, such as for presenting information or controllingvarious devices as described in more detail herein.

For example, optional sleep staging at processing 206, such as in asleep staging processing module may be implemented. However, any one ormore of such processing modules/blocks may optionally be added (e.g.,sleep scoring or staging, subject recognition processing, fatiguerecognition, sleepiness recognition, presence absence recognition,biometric identification of a person (biometric recognition), motionmonitoring and/or prediction processing, appliance control logicprocessing, or other output processing, etc.). In some cases, thefunctions of signal post-processing 206 may be performed using any ofthe components, devices and/or methodologies of the apparatus, systemand method described in any of the following patents or patentapplications, wherein the entire disclosures of each is incorporated byreference herein: International Patent Application No.PCT/US2007/070196, filed Jun. 1, 2007 and entitled “Apparatus, System,and Method for Monitoring Physiological Signs;” International PatentApplication No. PCT/US2007/083155, filed Oct. 31, 2007, entitled “Systemand Method for Monitoring Cardio-Respiratory Parameters;” InternationalPatent Application No. PCT/US2009/058020, filed Sep. 23, 2009, entitled“Contactless and Minimal-Contact Monitoring of Quality of LifeParameters for Assessment and Intervention;” International ApplicationNo. PCT/US2010/023177, filed Feb. 4, 2010, entitled “Apparatus, System,and Method for Chronic Disease Monitoring;” International PatentApplication No. PCT/AU2013/000564, filed Mar. 30, 2013, entitled “Methodand Apparatus for Monitoring Cardio-Pulmonary Health;” InternationalPatent Application No. PCT/AU2015/050273, filed May 25, 2015, entitled“Methods and Apparatus for Monitoring Chronic Disease;” InternationalPatent Application No. PCT/AU2014/059311, filed Oct. 6, 2014, entitled“Fatigue Monitoring and Management System;” International PatentApplication no. PCT/EP2017/070773, filed on Aug. 16, 2017, entitled“Digital Radio Frequency Motion Detection Sensor”; International PatentApplication No. PCT/AU2013/060652, filed Sep. 19, 2013, entitled “Systemand Method for Determining Sleep Stage;” International PatentApplication No. PCT/EP2016/058789, filed Apr. 20, 2016, entitled“Detection and Identification of a Human from Characteristic Signals;”International Patent Application No. PCT/EP2016/080267, filed on Dec. 8,2016, entitled “Periodic Limb Movement Recognition with Sensors”;International Patent Application No. PCT/EP2016/069496, filed 17 Aug.2016, entitled “Screener for Sleep Disordered Breathing;” InternationalPatent Application No. PCT/EP2016/058806, filed on Apr. 20, 2016,“Gesture Recognition with Sensors”; International Patent Application No.PCT/EP2016/069413, filed Aug. 16, 2016, entitled “Digital Range GatedRadio Frequency Sensor,” International Patent Application No.PCT/EP2016/070169, filed Aug. 26, 2016, entitled “Systems and Methodsfor Monitoring and Management of Chronic Disease;”, International PatentApplication No. PCT/US2014/045814, filed Jul. 8, 2014, entitled “Methodsand Systems for Sleep Management;” U.S. patent application Ser. No.15/079,339, filed Mar. 24, 2016, entitled “Detection of PeriodicBreathing.” Thus, in some examples, the processing of detected movement,including for example, the breathing movement, may serve as a basis fordetermining any one or more of (a) a sleep state indicating sleep; (b) asleep state indicating awake; (c) a sleep stage indicating deep sleep;(d) a sleep stage(s) indicating light sleep (e.g., N1 or N2 simply lightsleep); and (e) a sleep stage indicating REM sleep. In this regard,while the sound and/or infrared related sensing technologies of thepresent disclosure provide for different mechanisms/processes for motionsensing such as using a speaker and microphone and processing of thesound signals, when compared to radar or RF sensing technologies asdescribed in some of these incorporated references, once a motion orbreathing signal, such as breathing rate is obtained with the soundsensing/processing methodologies described in this specification) theprinciples of processing breathing or other physiological movementsignals for an extraction of sleep states/stages information may beimplemented by the determination methodologies of these incorporatedreferences. For example, once the respiration rate and movement andactivity counts are determined from motion whether by RF or SONAR, sleepstaging is a common analysis. By way of additional example, the sensingwavelengths may be different between an RF pulsed CW and a SONAR FMCWimplementation. Thus, velocity may be determined differently such as bydetecting movement across a range (different sensing distances). ForFMCW, movement detection may be made at multiple ranges. Thus, one ormore moving targets may be tracked (whether it is two people, or indeeddifferent parts of a person —depending on their angle with respect tothe SONAR sensor).

Typically, an audio signal from a speaker, such as an in-vehicle audioentertainment/navigation or vehicle control system, may be generated andtransmitted towards a user for sensing within a vehicle, such as anaudio signal using one or more tones described herein. A tone providespressure variation in the vehicle (e.g., air) at one or more particularfrequencies. For purposes of this description, the generated tones (oraudio signals or sound signals) may be referred to as “sound”,“acoustic” or “audio” because they may be generated in a like manner toaudible pressure waves (e.g., by a speaker). However, such pressurevariations and tone(s) should be understood herein to be either audibleor inaudible, notwithstanding their characterization by any of the terms“sound”, “acoustic” or “audio.” Thus, the audio signal generated may beaudible or inaudible, wherein the frequency threshold of audibilityacross the human population varies by age. The signal may besubstantially inaudible such that most people cannot discern the sound(e.g., in the range above 18 kHz). The typical “audio frequency”standard range is around 20 Hz to 20,000 Hz (20 kHz). The threshold ofhigher frequency hearing tends to reduce with age, with middle agedpeople often unable to hear sounds with frequencies above 15-17 kHz,whereas a teenager may be able to hear 18 kHz. The most importantfrequencies for speech are approximately in the range 250-6,000 Hz.Speaker and microphone signal responses for typical consumersmartphones, or in-vehicle audio equipment, may be designed to roll offabove 19-20 kHz in many cases, with some extending to above 23 kHz andhigher (especially where the device supports a sampling rate of greaterthan 48 kHz such as 96 kHz). Therefore, for most people, it is possibleto use signals in the range of 17/18 to 24 kHz and remain inaudible. Foryounger people that can hear 18 kHz but not 19 kHz, a band of 19 kHz tosay 21 kHz could be employed. It is noted that some domestic pets may beable to hear higher frequencies (e.g., dogs up to 60 kHz and cats up to79 kHz). A suitable range for the sensing audio signal of the presenttechnology may be in a low ultrasonic frequency range such as 15 to 24kHz, 18 to 24 kHz, 19 to 24 kHz, 15 to 20 kHz, 18 to 20 kHz or 19 to 20kHz.

Any of the arrangements and methods of audio sensing, such as with useof low frequency ultrasonic sensing signals, as described inPCT/EP2017/073613 may be implemented by the processing device describedherein. However, in some cases, a dual tone FMCW (also referred to as adual ramp technology) may be implemented as described herein.

For example, a triangular FMCW waveform with one “tone” (i.e., that isswept up and down in frequency) may be generated by the processingdevice using its speaker(s) where the waveform has the frequency versustime characteristics illustrated in FIG. 4A, and where processing of theup-sweep, or just the down-sweep, or even both may be evaluated fordistance detection. The phase-continuous triangular form for one tone ishighly desirable as it minimizes or removes any audible artefact in theplayed sound created by a phase discontinuity. A ramp variant of thiscan give rise to a very unpleasant and audible buzzing sound, as thespeaker(s) is/are asked to jump from playing a certain amplitude soundat a frequency to a much lower (or much higher) frequency at a similaramplitude within the space of a sample; the mechanical change in thespeaker can give rise to a click, and the frequent repetition of thechirp means that the user hears a buzz (many closely spaced clicks).

Optionally, in some versions of the present technology, the sensingsignal (e.g., acoustic) as a FMCW may be implemented with special dual“tones” with a ramp waveform (e.g. which consists of an up-sweep or adown-sweep only)—so that there is a sharp change in frequency from theend of one ramp (frequency ramp up and down) to the next (frequency rampup and down) without audible artefact. Such a dual “tone” frequencymodulated waveform showing its frequency characteristics relative totime, where at least two changing frequency ramps overlap during aperiod of time and these frequency ramps each may have a differentfrequency relative to the other(s) at any instant of time in the periodsuch as for the duration of the ramping, is illustrated in FIG. 4B inrelation to the dashed line versus the solid line. This can ultimatelysimplify the data processing in the system, and also remove thepotentially high amplitude transition at each point of a triangularwaveform. Sharp and repetitive transitions can sometimes trigger strangebehavior in a system's low level DSP/CODEC/firmware.

FIGS. 4A and 4B show a frequency domain comparison of FMCW single tone(FIG. 4A) and dual tone (FIG. 4B) implementations. A single tone (FIG.4A) may preferentially include a downsweep (a reduction in producedfrequency over time) to ensure inaudibility. However, a downsweep may beomitted but may cause some audibility. A dual tone (tone pair) (FIG. 4B)can help avoid the need for such a downsweep, as the time domainrepresentation is shaped such as to be inaudible. FIG. 4B shows thefirst tone 4001 and the optional second tone 4002 overlap. The figuredoes not show the received echo (i.e., the reflection signal). Thus,tones form a first sawtooth frequency change overlapped with a secondsawtooth frequency change in a repeated waveform. They are continuoussuch that they may be repeated during a sensing period.

Thus, there are different ways that the acoustic sensing signal can becreated when implementing a low frequency ultrasonic sensing system withan FMCW type of approach. This may involve differences in waveform shapein the frequency domain (e.g., triangular (symmetric or asymmetric),ramp, sinusoidal etc.), period (duration of the “chirp” in time), andbandwidth (frequency covered by the “chirp”—e.g., 19-21 kHz). It is alsopossible to use two or more simultaneous tones in an FMCW configuration.

The choice of number of samples defines a possible output demodulatedsampling rate (e.g., 512 samples at a sampling rate of 48 kHz equates to93.75 Hz (48,000/512), whereas a 4096 sample duration sweep time equatesto 11.72 Hz (48,000/4096). If a triangular waveform is used with a 1500sample uptime, and 1500 sample downtime, then the output sampling rateis 16 Hz (48,000/3000). For this type of system, synchronization can beperformed by multiplying the signal by a reference template, forexample.

Regarding the choice of the output sampling rate, empirical testing hasshown that operating in the approximate region of 8 to 16 Hz ispreferable, as it broadly avoids 1/f noise (low frequency effects due toair movement, potentially strong fading, and/or room modes) as well asstaying out of the reverberation region seen at higher demodulatedsampling rates (i.e., we have allowed time for the energy in any onefrequency of sensing waveform “chirp” to fade before the next similarcomponent in next “chirp”). Presented another way, if you make bins toowide, changes in airflow and temperature (e.g., opening door and heatgoes in or out of room) means any block you are looking at could containan unwanted baseline drift which can look like breathing. Practically,this means that a wave is seen to move across the band (across rangebins) as the air moves. This is distinct from more localized effectsfrom a desk or pedestal fan, or an air conditioning or other HVACsystem. Effectively, if the blocks are made too wide, the system beginsto “look like” a CW system. On the other hand, one can get reverb if thesystem works at too high a refresh rate (i.e., too short a ramp).

For a triangular FMCW waveform with one “tone” (i.e., that is swept upand down in frequency) as illustrated in FIG. 4A, a system can process,for example, just the up-sweep, or just the down-sweep, or indeed bothmay be processed for distance detection. The phase-continuous triangularform for one tone is highly desirable as it minimizes or removes anyaudible artefact in the played sound created by a phase discontinuity. Aramp variant of this can give rise to a very unpleasant and audiblebuzzing sound, as the speaker(s) is/are asked to jump from playing acertain amplitude sound at a frequency to a much lower (or much higher)frequency at a similar amplitude within the space of a sample; themechanical change in the speaker can give rise to a click, and thefrequent repetition of the chirp means that the user hears abuzz (manyclosely spaced clicks).

An important consideration to implemented such a dual tone signal isthat the resulting shape is made (shaped) such that the speaker/systemdoes not need to make a sharp transition, and it has zero points. Thiscan reduce the need for filtering that would otherwise be implemented torender the signal inaudible. For example, high pass or band passfiltering may be avoided while still permitting the signal to operate asan inaudible sensing signal. The presence of zeros in the waveform easessignal processing because that the zeros simplifies synchronization ofthe transmit and the receiving of such a signal (e.g., fordemodulation). A consequence of the dual tones is that it offers anelement of fading robustness as more than one tone is used—and fadingcan vary with the frequency used, as well as phase or frequency (e.g.,one might use a 100 Hz offset between the FMCW tones in a dual tonesystem).

Performance of the FMCW single tone of FIG. 4A and the FMCW dual tone ofFIG. 4B may be considered in reference to FIGS. 8 and 9. FIGS. 8A, 8Band 8C show signal characteristics of the FMCW single tone example ofFIG. 7A. FIGS. 9A, 9B and 9C show the signal characteristics of the FMCWdual tone example of FIG. 7B.

FIG. 8A shows the transmitted (Tx) signal 8001, and the received (Rx)reflection 8001-R (echo) operating as a triangular single tone FMCWoperating in an acoustic sensing system. FIG. 8B shows the time domainwaveform. FIG. 8C shows the spectral content of the signal. As evident,there is still content at lower frequencies (outside the peak arearelating the bandwidth of the FMCW signal). Such lower frequencies maythus be in an audible frequency range and thereby resulting in anundesirable performance characteristic.

FIG. 9A depicts a dual tone ramp FMCW signal in signal graph 9002.Signal graph 9002 represents both tones, and signal graph 9002-Rrepresents the received echo of the two tones/multi-tone. FIG. 9B showsa cosine-like functional shape of the dual tone, with the zero points(resultant zero crossings). FIG. 9C shows a much smoother peak and lowerpower amplitude at lower frequencies. The slope region SR of FIG. 9C,when compared to the slope region SR of FIG. 8C, illustrates a sharperdecline in power (dB) of the dual tone ramp FMCW in/to the lowerfrequencies. The sharper roll-off from the range of the high(substantially inaudible, utilised for sensing) frequencies and into thelower (audible, not typically utilised for sensing) frequencies, is adesirable acoustic sensing property as it is less obtrusive for theuser. The power at lower frequencies (outside the peak area relating tothe bandwidth of the FMCW signal) can be 40 dB less than that in thecase of the single tone FMCW triangular form illustrated in FIG. 8C. Asillustrated in FIG. 9C, the upper smooth peak region PR of FIG. 9C whencompared to the multi-edged peak region PR of FIG. 8C, indicates thatthe dual tone ramp FMCW signal can have better acoustic sensingproperties and is less demanding on the speakers.

Such a multiple tone FMCW or dual tone FMCW system (for example runningon a Linux based single board computer) can provide sensing such that itis possible to identify multiple persons within the sensing range of 4 mor more. It can also detect heart rate for example at 1.5 meters fromthe processing device, and respiration rate(s) at out to approximately 4meters or more. An exemplar system could use two tones at 18,000 Hz and18,011.72 Hz, which could ramp to, for example, 19,172 Hz and 19183.72Hz respectively.

For this ramp of 1,172 Hz, we can consider using, for example, an FFT ofsize 4096 points, with bin width of 48,000 Hz/4096=11.72. For speed ofsound as 340 m/s, we note: 340 ms/s/11.72/2 (for out and back)=14.5 mover 100 bins or 14.5 cm for each bin. Each “bin” can detect up to oneperson (per bin) for example (but in practice persons would be separatedby more than this.) As part of a synchronization process, the signalcould be squared, for example, to avoid a more computationally expensivecorrelation operation, where the signal is multiplied by a referencetemplate. Independent of the FFT size used, the maximum range resolutionis speed-of-sound/(Bandwidth*2)=340/(1172*2)=14.5 cm. However, asynchronization process may optionally be provided that includescross-correlating a sensed reflected signal with a sensed direct pathsignal. A synchronization process may optionally include multiplying areference template with at least a portion of the sensed reflected soundsignal.

FIG. 5 illustrates an example of “self-mixing” demodulation of a dualtone FMCW ramp by multiplying the signal by itself (squaring).Optionally, demodulation may be carried out by multiplying the receivedecho signal with a signal representative of the generated transmitsignal (e.g., a signal from an oscillator) to produce a signalreflecting distance or motion in the range of the speaker or processingdevice 100. The processing produces a “beat frequency” signal which issometimes referred to as an “intermediate” frequency (IF) signal. WithFMCW, when the receive Rx signal is demodulated, such as by a localoscillator or by itself as described in more detail herein, and low passfiltered, it may produce an unusual “intermediate” signal that is notyet considered to be baseband. The IF signal may be processed, such asby application of fast Fourier transform processing (FFT), to becomebaseband (BB).

As illustrated in FIG. 5, the demodulation is conducted with the receive(reflected sound signal) Rx signal only. That is mathematically possiblebecause the Rx signal contains a large percentage of a signalrepresentative of the transmit (Tx) signal (e.g., the produced soundwhich may, in part, travel a direct path from the speaker to themicrophone and be sensed with the reflected sound) in it. The device canmultiply the receive signal Rx by itself (such as by just squaring itbecause demodulation can be considered a multiply operation). This canbe followed by a filtering process (e.g. lowpass).

Although FIG. 5 illustrates self-mixing, several different approachesmay be implemented to derive a motion signal with the reflected signal,and the sensing signal (i.e., Tx or sound signal). In one such version,a local oscillator LO (which may also produce the sound signal) caneffectively produce a copy of the Tx signal for demodulation. Theactually produced Tx signal might be slightly different than theinternal signal from the oscillator because of delay or distortion.Demodulation can then be conducted by multiplication of the signal fromthe local oscillator LO(Tx)*Rx which can also be followed by filtering(e.g., lowpass).

In another version, two local oscillators may be implemented to generatetwo LO signals. For example, a Sin and Cosine copy of the LO signal maybe implemented to provide for quadrature demodulation of the receivesignal. Typically, only one signal from an oscillator (either Sin orCosine) is transmitted. The exact Tx signal will be somewhat differentfrom the signal from the local oscillator LO due to delay or distortion.In this version, demodulation may be conducted (a) RX*LO(Sin) and (b)RX*LO(Cos), which may be followed in each case by filtering (e.g.,lowpass) to produce both I and Q demodulation components.

Sensing—Mixing (Coexistence) of Acoustic Sensing with Other AudioPlayback by the System (Music, Speech, Snoring Etc.)

Some versions of the present technology may be implemented when theprocessing device 100 may be using its speaker and/or microphone forother purposes, in addition to the ultrasonic sensing described herein.Additional processes may be implemented to permit such simultaneousfunctioning. For example, the transmit bitstream (acoustic sensingsignal) may be digitally mixed with any other audio content (audible)that is being played by the speaker as previously mentioned forsimultaneous audio content production and ultrasonic sensing. Severalapproaches can be used to carry out such audible audio content andultrasonic processing. One approach requires that the other audiocontent (which could be mono, stereo or many more channels—such as inmany channel surround sound systems) is preprocessed to remove anyspectral content that would overlap with the sensing waveform. Forexample, a music sequence might contain components of over 18 kHz whichwould overlap with, for example, an 18 to 20 kHz sensing signal. In thiscase, the music components near 18 kHz can be low pass filtered out. Asecond option is to adaptively filter the music so as to removefrequency components for the short periods of time during overlappingsensing (direct path and echo), and allow the unfiltered musicotherwise; this approach is designed to retain the fidelity of themusic. A third option may simply make no changes whatsoever to the musicsource.

It should be noted that where delays are deliberately added to audiosources on certain channels (e.g., Dolby Pro Logic, Digital, Atmos, DTSetc. or indeed virtualized spatializer functions), any such in-bandsignals are also accordingly processed, and the sensing waveform willeither not be delayed or the delay would be allowed for, when processingthe echoes).

Sensing—Coexistence with Voice Assistants

It should be noted that certain realizations of ultrasonic sensingwaveforms (e.g., triangular FMCW), may have an unintended and unwantedimpact on certain voice assistants that are performing voice recognitionservices, such as Google Home, as they have spectral content within theaudible band. Such potential cross talk can be avoided by using a dualramp tone pair, or pre-filtering (high pass or band pass filtering thetriangular chirp) the sensing waveform, or adapting the voicerecognition signal processing to be robust to the ultrasonic sensingsignal components.

Consider an FMCW ramp signal y as follows:

y=[A Cos(2pi(f ₁ +f ₂ t)t+phi]₀ ^(T)

This ramp from frequency f_1 to frequency f_2 in a time period T. Thishas sub harmonics as it is switched at a time period of T.

An analysis of this shows that it has out of band harmonics which appearat lower frequencies and so can be heard.

Now consider a specific dual ramp pair y as follows:

y=[A Cos(2pi(f ₁ +f ₂ t)t+phi]₀ ^(T)−[A Cos(2pi(f ₁+(1/T)+f ₂ t)t+phi]₀^(T)

Thus, the sub-harmonics are cancelled (subtracted in the above), and thesignal retained. The 1/T is very specific; by using (1/T), or indeed−(1/T), the effect of the switching at time period T is canceled out.Thus, the resulting signal is inaudible. It does this while beingmathematically simple, which is an advantage as it is notcomputationally onerous on a device (e.g., a smart mobile phone device).

Because the dual tone switches at DC level (“0”), there is a naturalpoint in the waveform chirp (a beginning and an end of the signal) toturn off, such as to avoid clicking (i.e., turn on and off in a way toavoid a loudspeaker making a big jump). The “0” 's also allow us tointroduce a quiet period between each chirp, or indeed between groups ofchirps, in order to mitigate reverberation—and/or to identify a specifictransmitter (i.e., to overlay a sequence of on/off chirp times).

The lack of sub-harmonics is also an advantage as it removes a possiblesource of interference when considering two devices operating in a roomat the same time. Thus, two different devices can use non-overlapping(in frequency) tone pairs—or indeed overlapping in frequency (but not intime—due to the addition of non-overlapping quiet periods) tone pairs.The latter can be an advantage where loudspeaker/microphone combinationshave limited available inaudible bandwidth (i.e., their sensitivityrolls off severely over 19 or 20 kHz).

Even comparing a relatively inaudible triangular FMCW signal to a dualtone ramp, the latter has a very much smaller level of sub harmonics(approaching the noise floor on a real world smart device—e.g., near thequantization level).

Because a dual tone ramp can be ramped up or down (rather thantriangular) and yet have no out of band components, there are no interramp bleed problems which can occur with a triangular ramp.

A standard ramp audio signal cannot be made inaudible without extensivefiltering, which would potentially distort the phase and amplitude ofthe resulting waveform.

Sensing—Calibration/Vehicle Mapping to Optimize Performance

The processing device may be configured with a set-up process. When thedevice is first set up (or periodically during operation) it can sendout an acoustic probing sequence to map the vehicle environment, thepresence and/or number of people in the vehicle etc. The process can berepeated if the device is subsequently moved, or the quality of thesensed signals is detected to have decreased. The system may also emitacoustic training sequences in order to check the capabilities of thespeaker(s) and mic(s), and estimate equalization parameters; real worldtransducers may have some non-linearities in the ultrasound frequenciesused by the system, as well as temperature and turn on characteristics(e.g., as a loud speaker may take several minutes to settle).

Sensing—Beam Forming for Localization

It is possible to implement dedicated beam forming or utilise existingbeam forming functionality—i.e., where signal processing is employed toprovide directional or spatial selectivity of signals sent to, orreceived from, an array of sensors. This is typically a “far field”problem where the wavefront is relatively flat for low frequencyultrasound (as opposed to medical imaging, which is “near field”). For apure CW system, audio waves travel out from the speaker, leading toareas of maxima and minima. However, if multiple transducers areavailable, it becomes possible to control this radiation pattern to ouradvantage—an approach known as beam forming. On the receive side,multiple microphones can also be used. This allows the acoustic sensingto be preferentially steered (e.g., steering the emitted sound and/orthe received sound waves where there are multiple speakers) in adirection, and swept across a region. For the case of a user in areclining seat or (travel bed), the sensing can be steered towards thesubject—or towards multiple subjects where there are, for example, twopersons in neighboring reclining seats (travel beds). Beam steering canbe implemented on transmit or receive. As low cost ultrasonictransducers (microphone or speaker) can be quite directional (e.g., fora small transducer, where the wavelength is comparable to the size ofthe transducer), this can restrict the area in which they can be steeredover.

Sensing—Demodulation and Down Conversion

Returning to FIG. 5, the sensed signal is demodulated, such as with themultiplier (mixer) module 7440 shown in FIG. 7 or according to thedemodulator of FIG. 5, to produce a baseband signal that may be furtherprocessed to detect whether there is “presence” in the sensing field—adisturbance in the demodulated signal that relates to a change in theechoes received, related to a characteristic motion of a person. Wherethere is a strong received “direct path” (high crosstalk from speaker tomicrophone, e.g., transmission through a solid versus through air and/orshort distance from speaker to mic) signal, in addition to the receivedecho signal, multiplication of the resulting sum can be performed todemodulate. Otherwise, the received echo can be multiplied (mixed) witha portion of the originally transmit signal, which is extracted in anelectronic, and not acoustic, form. In this specific example, the systemis not multiplying the receive signal by the transmit signal todemodulate it (although it may other embodiments). Instead, the systemmay multiply the receive signal (which contains an attenuated version ofthe transmit signal, as well as the receive echo(es)) by itself asfollows:

Transmit=A _(TX)(Cos(P)−Cos(Q))

Receive=A(Cos(P)−Cos(Q))+B(Cos(R)−Cos(S))

Selfmixer=[A(Cos(P)−Cos(Q))+B(Cos(R)−Cos(S))]×[A(Cos(P)−Cos(Q))+B(Cos(R)−Cos(S))]i.e., receive×receive

Self Mixer components (Demodulated) after low pass filtering:

$0.5x\begin{matrix}{{AA}\mspace{11mu} {{Cos}( {P\text{-}P} )}} & {{- {AA}}\mspace{11mu} {{Cos}( {P\text{-}Q} )}} & {{AB}\; {{Cos}( {P\text{-}R} )}} & {{- {AB}}\; {{Cos}( {P\text{-}S} )}} \\{{- {AA}}\mspace{11mu} {{Cos}( {Q\text{-}P} )}} & {{AA}\mspace{11mu} {{Cos}( {Q\text{-}Q} )}} & {{- {AB}}\mspace{11mu} {{Cos}( {Q\text{-}R} )}} & {{AB}\mspace{11mu} {{Cos}( {Q\text{-}S} )}} \\{{BA}\mspace{11mu} {{Cos}( {R\text{-}P} )}} & {{- {BA}}\mspace{11mu} {{Cos}( {R\text{-}Q} )}} & {{BB}\mspace{11mu} {{Cos}( {R\text{-}R} )}} & {{- {BB}}\mspace{11mu} {{Cos}( {R\text{-}S} )}} \\{{- {BA}}\mspace{11mu} {{Cos}( {S\text{-}P} )}} & {{BA}\mspace{11mu} {{Cos}( {S\text{-}Q} )}} & {{- {BB}}\mspace{11mu} {{Cos}( {S\text{-}R} )}} & {{BB}\mspace{11mu} {{Cos}( {S\text{-}S} )}}\end{matrix}$

Self Mixer Output (Demodulated) after equation simplification:

$\begin{matrix}{AA} & {{- {AA}}\mspace{11mu} {{Cos}( {P\text{-}Q} )}} & {{AB}\mspace{11mu} {{Cos}( {P\text{-}R} )}} & {{- {AB}}\mspace{11mu} {{Cos}( {P\text{-}S} )}} \\{{- {AB}}\mspace{11mu} {{Cos}( {Q\text{-}R} )}} & {{AB}\mspace{11mu} {{Cos}( {Q\text{-}S} )}} & {{- {BB}}\mspace{11mu} {{Cos}( {R\text{-}S} )}} & {BB}\end{matrix},$

where AA and BB are DC components

Demodulated components that contain reflected signal information (can bestatic as well as movement related):

−AB Cos (Q-R) AB Cos (Q-S) AB Cos (P-R) −AB Cos (P-S)

The advantages of this are: no synchronization is required betweentransmit and receive, as all timing information is contained in thereceive only, and it is computationally fast and simple (square anarray).

After I, Q (in phase and quadrature) demodulation, there is a choice ofhow to separate the low frequency components relating to air turbulence,multi-path reflections (including fading related to same) and other slowmoving (generally non-physiological) information. In some cases, thisprocessing can be called clutter removal. The DC level (mean) can besubtracted, or some other detrending (such as linear trend removal)performed on an overlapping or non-overlapping block basis; a high passfilter can also be applied to remove DC and very low frequencycomponents (VLF). The “removed” information can be processed to estimatethe intensity of such DC and VLF data—such as whether there are strongair currents, or significant multipath effects. The filtered demodulatedsignal can then be passed to a spectral analysis stage. The other choiceis not to use high pass filters and to pass the unfiltered signaldirectly to the spectral analysis processing block, and carry out the DCand VLF estimation at this stage.

Coexistence of Different Sensing Devices/Applications

It can be seen that coded or uncoded ultrasonic signals may be generatedby different devices to permit devices and systems to implementidentification and other data interchange purposes. For example, amobile phone application may be configured to generate such signals forcommunication purposes in order to identify itself to another sensingenabled device/system in its proximity, such as a smart infotainmentsystem of a vehicle and vice versa. These types of signals may be usedin place of short range radio frequency communication (e.g., whereBluetooth is not available or is disabled) for identification. Thedevice of the system can then automatically determine existence of otherprocessing devices in the sensing vicinity (e.g., via inaudibleacoustically generated communication signals from another processingdevice) and adjust the parameters of the generated sensing signals sothat they can operate in non-interfering sensing modes (e.g., by usingdifferent frequency bands and/or not overlapping in time).

Example System Architecture

Exemplar system architecture of a voice enabled sleep improvement systemusing low frequency ultrasonic biomotion sensing is illustrated in FIG.6 in vehicle 1 l 1. The system may be implemented with the sensingtechniques described herein (e.g., multi-tone FMCW acoustic sensing). Auser can talk to a voice activated system such as the vehicularprocessing device 100 that was previously activated to monitor theuser's sleep. For example, a verbal instruction can query the system tomonitor the vehicle and produce an audible report of determinedsleepiness, sleep score, respiratory (SDB) events, health condition orother motion related statistics. Based on the report, the processing ofthe system can also produce audible warnings, advice or other controlsignals for other devices.

System processing for detection of motion in a vicinity of aspeaker-enabled processing device 100 that is enabled with the lowfrequency ultrasonic sensing of the present technology may be consideredin relation to the example modules illustrated in FIG. 7. The processingdevice 7102 includes a speaker(s) 7310 and, optionally, microphone(s)7302 as well as a microcontroller 7401 with one or more programmableprocessors. The modules may be programmed into a memory of themicrocontroller such as of an in-vehicle audio equipped system. In thisregard, an audio sample or audio content may be upsampled by optionalupsampling processing module at 7410 and may be provided to a summermodule 7420, such as if optional audio content is produced by one ormore speaker(s) simultaneously with the sensing signal. In this regard,the summer module 7420 optionally combines the audio content with theFMCW signal in the desired frequency ranges from the FMCW process module74430 that produces the FMCW signal (e.g., the dual tone FMCW signal inthe desired low ultrasonic frequency ranges). The summed FMCW signal maythen be processed such as by a converter module for output by thespeaker 7310. The FMCW signal is also applied to a demodulator such as amultiplier module 7440 where the FMCW signal is processed (e.g.,mixed/multiplied) with the received echo signal observed at themicrophones 7302. Prior to such mixing, the received echo signal may befiltered, such as adaptively, as previously mentioned herein to removeundesired frequencies outside the frequency spectrum of interest, suchas frequencies associated with the operations of the motor vehicle(e.g., motor vibration or wind sound). An audio output processingmodule(s) 7444 may optionally down sample the filtered output and/orconvert the signal to produce an audio signal. The demodulated signaloutput from the multiplier module 7440 may then be further processed,such as by post-processing module 7450. For example, it may be processedby frequency processing (e.g., FFT) and digital signal processing toimprove the raw motion signal detected or otherwise separate motions byfrequency range so as to isolate (a) respiration motion or movement, (b)cardiac motion or movement and (c) gross motion or movement, such asgross body motion or gross body movement, for example. The physiologicalmovement signal(s) may then be recorded or otherwise processed, e.g.,digitally, by characteristics processing at 7460 to characterize variousmotions of the signal so as to detect various informational output aspreviously mentioned (sleep, sleepiness, fatigue, sleep stage, motion,respiration events, etc.).

In relation to detection of gross movement or gross body motion, suchmovement may include any of arm movement, head movement, torso movement,limb movement, and/or whole-body movement, etc. Methodologies for suchdetections from transmitted and reflected signals for motion detection,which may be applied to SONAR sound-type motion detection, may beconsidered and applied, for example, as described in InternationalPatent Application Nos. PCT/EP2016/058806 and/or PCT/EP2016/080267, theentire disclosures of which are incorporated herein by reference. By thenature of it, such RF or SONAR technology may be seeing all bodymovement at once, or at least most of it—and it may depend on whereexactly the “beam” is directed. For example, is it illuminatingprimarily the head and chest, or the whole body etc. Leg movement, suchas when it is periodic, may primarily distinguished as a motion based onfrequency of movement, and optionally by performing different automatedgain control (AGC) operations. Respiration detection is most effectivewhen there is less gross body movement, to isolate the characteristicfrequencies and signal shape of a breathing waveform (either normal,COPD or CHF changes to rate over time and inspiration/expiration ratio,SDB events, longer term SDB modulation etc.)

When motion is associated with a person in bed, the largest amplitudesignals may be associated with a full body movement, such as a roll. Ahand or leg movement may be faster (e.g., velocity from I/Q signal) butlower relative amplitude. Thus, different components, and or sequencesof components, of such a movement by analysis of a motion signal may beconsidered in the identification such as whether it starts with grossmovement and acceleration, velocity of arm movement, then stops, etc.This identification may be more targeted for different motion gestures.

Low Frequency Ultrasonic (SONAR) Sensing—for in-Vehicle Use

Many vehicles contain audio devices that are capable of emitting andrecording sounds in the low frequency ultrasonic range at just above thehuman hearing threshold—e.g., vehicle infotainment systems. Such devicesand systems can be adapted to perform physiological sensing of thevehicle's occupants using low frequency ultrasonic techniques. Suchsensing may be performed without impacting the original intendedfunctions of the standard audio systems. In one example, such a sensingfunctionality can be implemented by way of a software update (i.e.,allowing additional useful functionality to be added without increasingthe cost of goods). In some cases, one or more of the transducers in anew device or system may be specified to support the audio frequencyranges for low frequency ultrasonic sensing, with additional testing atmanufacture to ensure they meet this specification.

Such acoustic (either audible or inaudible) sensing technology can beused for a wide variety of purposes including pro-active healthmanagement, medical devices, and security functions.

A low frequency ultrasonic system operating up to around 25 kHz can berealized on a mobile smart device or smart speaker device. Thistransmits sound energy towards one or more subjects using one or moretransducer on the electronic device, where the transducer is configuredto generate sound energy over a range of frequencies that includesfrequencies less than 25 kHz. The speakers could be contained in a smartphone, a smart speaker, a sound bar, a portable TV screen, or many otherdevices and configurations that contain transducers capable ofsupporting low frequency ultrasonic sensing and processing. Many modernvehicles include onboard processors/computers that can control many ofthe car instruments and functionalities. If such a computer isimplemented to control the speaker system of the car or other speakerswithin the car, it effectively creates a smart speaker system. Whilstaspects of this specification in some instances identifies the vehicleas a car, it is understood that the sensing (e.g., sonar) principals andconsiderations for a car may be applied to biometric sensing in othertypes of vehicles, such as trucks, busses, trains, airplanes, boats etc.

Audible sounds such as the sound of breathing, coughing, snoring whenasleep, gasping, wheezing, speech, sniffing, sneezing can be extractedand classified from the sensed audio signal within the vehicle so as topermit isolation of these sounds from the reflected sensing signal thatwill be detected for motion sensing. Some of these sounds (e.g., acough) can mask the sensing signal (especially if it is operating at avery low sound pressure level), which is not desirable. However, suchsounds may still be detectable so that they can be separated from otherenvironmental sounds (e.g., a car horn blowing, motor noise, streetsounds, wind, a slamming or closing door etc.). The sound of breathingis typically of better signal quality in a quiet environment, and canprovide a good second estimate of inspiration/expiration time (and thusbreathing rate) when complimented with an active sensing approach suchas SONAR or RADAR (including an RF one) (which are primarily detectingtorso and limb movements), or camera/infra-red systems. In other words,the system can still extract information about the characteristics ofsounds, even very loud sounds mean that we need to skip small sectionsof the sensed signal as the associated signal quality drops below anacceptable threshold.

In SONAR systems, the air movement caused by an inspiration orexpiration may also be detected by methods that track the resultingtravelling wavefront (due to the disturbance of acoustic modes set up inthe sensing environment—if the sensing signal persists long enough toexperience reverberation). Detecting snoring directly from the audiblesignature is easier, as it be a relatively loud process for example, byusing a running mean filtered maximum decibel level (i.e., max of afiltered section of a signal) to classify snoring as mild (40-50 dB),moderate (>50-60 dB), or severe (>60 dB). (see Nimrod Maimon, Patrick J.Hanly, “Does Snoring Intensity Correlate with the Severity ofObstructive Sleep Apnea?” J Clin Sleep Med. 2010 Oct. 15; 6(5):475-478.)

Thus, in some cases, the processing device 100 may use motion detection(e.g., Sonar) techniques for detecting respiration. However, in somecases, acoustic analysis, of an audible breath signal at the microphone,may be implemented by the processing device 100 for the detection ofrespiration.

RF (RADAR) Sensing

Some existing car alarm systems may include single pulse Doppler RADARmodules for simple interior movement detection, especially in soft-topvehicles (convertibles) and/or sports cars. These may be enhanced (withupdated software) or replaced with modules that can localize motiondetection to specific areas of the vehicle —particularly to be able todetect and distinguish a person on each seat/sitting area. The sensormay be enhanced with technologies such as ultrawideband (UWB) sensingsignals or frequency modulated continuous wave (FMCW) sensing signal orincluding other coding schemes such as OFDM, PSK, FSK etc. in theirgenerated sensing signals. These can be implemented with a sensor havingan accurate ranging ability (1 cm or less). Such a sensor may sense in adefined area (e.g., set via the antenna design that may be configuredwithin the vehicle to have a particular seat oriented sensingdirection). In some cases, multiple antennas may be implemented for aparticular sensing area and may be used with beamforming techniques toset the distance sensing differences associated with the differentantennas. Multiple sensors can be used in a vehicle to provide coverageof multiple areas that a person (or pet) could be in (e.g., a sensor foreach seat).

Multimodal Data Processing

When using SONAR, RF, or infra-red detectors, a processing device 100may receive additional data or signals generated by equipment of thevehicle (e.g., to estimate occupancy) so that biomotion sensing may bebased on data from such equipment. For example, seat load sensors thatdetect whether a person sitting on a given seat may provide thebiomotion processing device 100 with information to determine when toinitiate biomotion sensing with respect to sensing that may beassociated with a particular seat. Similarly, seatbelt sensor(s) maydetect whether a person has clipped the seatbelt, or whether a childseat is fitted, so as to provide an indication of user presence for userrelated biomotion sensing. An infra-red system may optionally, forexample, be incorporated with a camera system that can track human eyemovement, such as for sleepiness detection. Other automobile or vehiclesensors may also be implemented to provide data regarding vehiclecharacteristics to the processing device for assisting with sensing orgenerating output. For example, the car's velocity from a speedometerand/or engine revolutions (RPM) from a tachometer may provide theprocessing device 100 information for filtering sensed signals, such asto remove car related noise (e.g., engine noise or wind noise etc.).

The processing device may be configured with distance information forevaluating relevant ranges/distances for detection of biomotioncharacteristics. For example, the processing device 100 may have adistance mapping (map) of a vehicle interior—such as of car. Such a mapmay be provided initially, e.g., at the design stage, to specify aninitial sensing configuration. Optionally, the sensing system, undercontrol of the processing device, may dynamically update (or detect) amap of the vehicle (e.g. cabin) when in use by one or more persons. Aninitial configuration may, for example, capture/detect the position ofseats, and most likely seat configurations (such as in a standard 5 seatcar, a minivan, a recreational vehicle etc.); where seats are movable,sensors can report the current settings to the system to update thesensing parameters (e.g., the position of a person sitting could movewith respect to a sensing loudspeaker in a car, as seat slides backwardsor forwards, or is folded down etc.).

For the monitoring of the interior of a shipping truck, the location of,for example, pallets of equipment or packages, shelving etc., processingdevice may include an automatic reconfiguration process with the sensorsin order to reject static objects from the analysis, and provide sensingin any free space where a person might hide. For the monitoring of theinterior of an ambulance or other patient transportation vehicle,processing device may include an automatic reconfiguration given themoveable nature and location of a stretcher. The interior of a caravanor trailer could also be monitored.

Biometric Feature Detection—Respiration, Cardiac, Movement, and Range

Processing Sensor Signals

The system, including particularly processing device 100, may receivedemodulated signals from a sensor (such as from SONAR, RF/RADAR, orinfra-red) such as optionally if demodulation is not performed by theprocessing device. The processing device 100 may then process the signalby separating components of interest such as direct current signals DCand very low frequencies VLFs (e.g., air currents), respiration, heartrate and gross physiological movement signals. These can beestimated/detected by bin searching in fast Fourier transform (FFT)windows, and tracking across windows, and/or via direct peak/trough orzero crossings analysis of a time domain signal at a specified range(e.g., a “time domain” signal for a specified distance range extractedusing a complex FFT analysis of the demodulated signal). This issometimes referred to as “2D” (two dimensional) processing as an FFT isperformed of an FFT such as described in International PatentApplication PCT/EP2017/073613.

For SONAR sensing, significant other information can be found in theaudio band and picked up by the microphone. Such information may beinfotainment sounds (music, radio, TV, movies), phone or video calls(including human speech), vehicle noise while in motion or stopped,ambient noise, and other internal and external sounds. Most of theseaudio components can be considered to be interferers, and may besuppressed (e.g., filtered) from biometric parameter estimation.

For RADAR sensing, signal components from other RF sources such as othervehicles (e.g., for interior biometric detection of people, or rangingfunctions such as for adaptive cruise control, collision avoidance,autonomous driving etc.) may be suppressed.

For infra-red sensing (such as when carrying out physiological sensingin addition to eye tracking), temperature changes and sun position maycause interference and may be taken into account. Thus, temperaturesensors, such as from a vehicle temperature sensor, and time may beevaluated in processing the sensing signal.

Regardless of the exact sensing technique used (RF, IR, SONAR), thereceived time domain demodulated reflected signal can be furtherprocessed (e.g., by bandpass filtering with a bandpass filter, evaluatedby an envelope detector, and then by a peak/trough detector). Envelopedetection may be performed with a Hilbert transform or by squaring therespiratory data, sending the squared data through a low-pass filter,and calculating the square root of the resulting signal. In someexamples, the respiratory data may be normalized and sent through a peakand trough detection (or alternatively a zero crossing) process. Thedetection process may isolate the inspiration and expiration portions,and in some cases, may be calibrated to detect the user's inspirationand expiration portions.

The respiratory activity is typically in the range 0.1 to 0.7 Hz (6breaths per minute—such as arising from paced deep breathing to 42breaths per minute—an atypically fast breathing rate in adults). Thecardiac activity is reflected in signals at higher frequencies, and thisactivity can be accessed by filtering with a bandpass filter with a passband of a range from 0.7 to 4 Hz (48 beats per minute to 240 beats perminute). Activity due to gross motion is typically in the range 4 Hz to10 Hz. It should be noted that there can be overlap in these ranges.Strong (clean) breathing traces can give rise to strong harmonics, andthese need to be tracked in order to avoid confusing a breathingharmonic with the cardiac signal. At longer distances from thetransducer (e.g., several meters), it can be very challenging to detectthe relatively small cardiac mechanical signal, and such heart rateestimation is better suited to settings where the user is lying quietlywithin a meter of the smart speaker—such as on a chair/couch or in bed.

Once absence/presence has been determined as “presence”, an estimate ofthe respiration, cardiac, and motion/activity signals (as well as theirrelative position and velocity if moving—such as getting in and out ofthe vehicle) is performed for one or more persons in the field of thesensor. It can be seen that a system that yields ranging information iscapable of separating the biometric data of multiple persons—even if themultiple persons have similar resting breathing rates (which is notuncommon in young couples).

Based on these parameters, it is possible to prepare a variety ofstatistical measures (e.g., average, median, 3^(rd) and 4^(th) moments,log, square root etc.), wave shape (morphological processing), and thensupply to a characterization system, such as a simple classification orlogistic regression function, or a more complex machine learning systemusing neural networks or artificial intelligence system. The purpose ofthis processing is to gain further insights from the gathered biometricdata.

Examples of these insights can be roughly characterized as securityrelated, consumer health related, and medical in nature.

Sleep Staging Analysis

As absence/presence/wake/(NREM) sleep stage 1/sleep stage 2/sleep stage3 (slow-wave sleep SWS/deep)/REM has a sequence related to theunderlying sleep architecture representing sleep cycles, it can behelpful to consider this as a sequenced rather than un-sequenced problem(i.e., reflecting typical sleep cycles, where a person remains in onestate for a period of time). The sequence of sleep imposes an explicitorder on the observations throughout for example the night (a “sleep”).

Some systems may also take advantage of knowledge of a “normal” sleeppattern having a more (higher prevalence) deep sleep (SWS) towards thebeginning of the night, and more REM sleep towards the end of the night.This prior knowledge, which could be used to weight (e.g., adjust priorprobabilities of these states over time) a classification system fornormal sleepers; however, it should be noted that these assumptions frompopulation normative values may not hold for non-normal sleepers, orthose that regularly nap during the day—or have poor sleep hygiene (poorsleep habits —such as widely varying ‘to-bed’ and ‘out-of-bed’ times).

Classically, sleep staging has been considered in 30 second “epochs”dating back to the Rechtschaffen & Kales guidelines (Rechtschaffen andKales, 1968) (a manual of standardized terminology, techniques andscoring system for sleep stages of human subjects. U.S. Public HealthService, U.S. Government Printing Office, Washington D.C. 1968) whichwhen looking at electroencephalogram EEG found a 30 second intervalideal for viewing alpha and spindles as a paper speed was 10 mm/s (onepage equates to thirty seconds). Of course, the real physiologicalprocess of sleep and wakefulness (and absence/presence) will not evenlysplit into 30 second blocks, so a longer or a shorter time can beselected. The system outlined here preferentially uses a 1 second (1Hertz) sleep stage output, although it uses longer blocks of data in anoverlapping fashion to deliver an update every 1 second (1 Hertz) (withan associated delay related to the size of the underlying processingblock). This 1 second output is used in order to better show subtlechanges/transitions in the sleep cycle.

Sleep Features—Manually Versus Automatically Generated

The sensed signal (a signal representing distance verses time (motion)information) is used to calculate various features, such as sleepfeatures. These features can then be used to derive informationregarding the user's physiological state.

For feature generation, a number of approaches may be implemented. Forexample, a human expert can manually produce features from the processedor unprocessed signals based on their experience, by looking at therespiratory and other physiological data and its distributions,understanding the physiological basis of particular changes, and trialand error. Alternatively, a machine can “learn” the features with somehuman supervision (a core concept of the field of “machine learning”)where labeled data with the expected outcome is supplied and some humanhelp provided, or in a fully automatic way where some or no labeled datamay be supplied.

Deep learning can be broadly considered in the following broadcategories: deep neural nets (DNN), convolutional neural nets (CNN),recurrent neural nets (RNN), and other types. Within DNNs, one canconsider deep belief networks (DBN), multilayer perceptron (MLP), aswell as stacked auto-encoders (SAE).

A Deep Belief Network (DBN) possesses a generative capability, e.g., toautomatically generate features from input data. Another approach forthis purpose is Fuzzy C-Means clustering (FCM), a form of unsupervisedlearning that aids finding the inherent structure in pre-processed data.

Handcrafted features can be formed by applying digital signal processingtechniques to sensed movement data. A respiration signal in an idealcase is perfectly sinusoidal with two amplitudes (deep or shallow) and aconstant frequency (constant breathing rate), described as you breathein and then out. In the real world, it can be far fromsinusoidal—especially as detected from the torso area via and acousticor radio frequency based sensing approach. For example, an inspirationmay be sharper than an expiration, and faster, and there may be a notchon the waveform if breath is held for a moment. The inspiration andexpiration amplitudes, as well as the respiration frequency, may vary.Some extraction methods focus on detecting the peak and trough, thendetecting the better quality of the two (say detecting a local peak anddiscarding the trough). This is not ideal if both the peak and thetrough times are needed in order to estimate both inspiration andexpiration times, as well as volumes (e.g., calculated by integration ofthe time domain signal versus a calculated reference baseline)—but canbe good enough for a respiration rate estimate.

Various methods can be used to assist in the estimate of any of thesefeatures, such as the respiratory and/or heart rate or amplitude.

For example, a peak and trough candidate signal extraction requiresrecovering the respiratory wave shape from noise (and there can be avariety of out-of-band and in-band noise, usually with a preponderanceof lower frequency noise which can complicate the accurate detection oflower breathing rates (e.g., 4-8 breaths per minute, which while unusualfor spontaneous breathing, can arise if a user is asked to guide theirbreathing to slower rates). Time domain detection methods include maxand min detection after low pass filtering, using an adaptive threshold(that adjusts over a block of multiple breaths to allow deep and shallowbreaths to be detected). Optionally, the signal may be low pass filteredand differentiated (e.g., a derivative function). Peaks in thedifferentiated signal relating to max rate of change may then bedetected to provide an indication of breath event. Such a methodextracts fiducial points of a respiratory waveform that is modeled as asinusoid with some noise on it. The LPF removes higher frequency noise.Differentiation is then done and peaks are detected. In effect, thisfinds points of maximum rate of change of the original signal, ratherthan the peaks and troughs of the original signal—as a respiratorywaveform is often clearest at maximum rate of change rather than a say awide peak (for example, if there is a breath hold for a short amount oftime). A potentially more robust method is to detect zero crossings(around a fixed or adaptive baseline), as the crossings of this boundaryis not directly impacted by local changes in the amplitude of thesignal.

While respiration signals may be easily visible in the time domainsignal (depending on the distance and angle of the chest from thesensor(s)), the cardiac motion is typically a very small signal incomparison to respiration. Higher order harmonics of respiration (e.g.,related to the wave shape) can complicate the cardiac signal extraction,and need to be rejected, or detected and excluded.

Frequency domain methods can also be applied, for example to therespiratory data. These methods can include using a detected peak in aband of an FFT (which may be windowed to combat spectral leakage) usinga block of data that may be overlapped (e.g., a block of 30 s of data ofa data stream that is repeatedly shifted by for example, one second) ornon-overlapped (e.g., the data stream is considered to benon-overlapping in thirty second chunks). A power spectral density PSDusing Welch's method, or a parametric model (autoregressive) may also beused, with a subsequent peak search. A spectral peak will tend to bewider (more spread) as the respiration signal becomes less sinusoidal,and can include harmonics if the shape has sharp peaks, sharp troughs,or notches. Another method is to use autocorrelation (describing thesimilarity of a signal to a shifted version of itself), where anassumption is that the underlying respiration wave shape is relativelystable for a period of time, and a periodic local maxima in theautocorrelation can be tracked and filtered by most likely candidate(e.g., not related to noise) maxima in order to estimate breathing rate.Autocorrelation can be carried out in the time domain, or by FFT in thefrequency domain. Time frequency approaches, such as wavelets are alsouseful where a suitable wavelet with a sinusoidal shape are selected(e.g., symlet, Debauchies etc.), that can perform strong de-noising;again, a peak detection is ultimately performed at the time scale ofinterest (i.e., within the target breathing rate range).

A Kalman filter (a recursive algorithm) can be applied to the timedomain signals to estimate the system state; this approach provides away to predict a future unknown state of a system, based only on the useof the preceding step. In addition to filtering, it can provide signalseparation, such as of large movement, respiration, and cardiacmovements.

In-Vehicular Sensing (Noise Contaminated Observations (e.g., forin-Vehicular Sensing, but Also Applicable to Detecting PhysiologicalMovement in Other Noisy Environments))

Any detection of respiration peaks and troughs needs to be aware ofpotentially confounding effects, such as the subject making a largemovement (such as rolling in bed or moving while driving), if thesubject stops breathing (e.g., an apnea) or exhibits very shallowbreathing (e.g., a hypopnea). Using sensing that can track locationprovides a useful means of separating these effects. For example, a rollcan be seen as both a high frequency movement, as well as change inlocation in space. Therefore, subsequent breaths may be higher or lowerin amplitude—but still be “healthy” breaths. In other words, thedetected amplitude change may be due to a change in the extractedreceived respiration signal (after down-conversion etc.) strength,rather than a change in the person's breathing. Therefore, it can beseen that this can allow a novel calibration approach, where thedetected distance can be used to relate signal strength to depth ofbreathing (and hence approximate tidal volume). Where no such movementor displacement is seen, a diminution, cessation, or change (e.g., dueto paradoxical movement on chest and abdomen during an obstructiveevent) of a specified duration range can be identified as abnormalbreathing (e.g., an apnea/hypopnea event).

It can be seen that a practical, robust cardiorespiratory estimationsystem can rely on multiple methods to localize the parameters. For goodsignal quality cases, a frequency (or time frequency) estimate canlocalize the likely breathing rate, an estimate of local breathingvariability, then extract subtle peak and trough times, and performcalibration with range in order to estimate inspiration and expirationvolumes (useful features for sleep staging). Such a signal qualitymetric is expected to vary over time. If there is a variation in themeasured breathing rate, the processing can be done over different timescales, e.g., averaging or median filtering over 30, 60, 90, 120, 150seconds etc.

In the SONAR case, the envelope of the raw received waveform (e.g., ofan acoustic FMCW signal) can be processed as a main, or as a secondaryinput such as when other additional sensing signals are implemented, forrespiration rate estimation (such as for using SONAR to provide extrainformation for an RF sensing system or vice versa). This is based onthe property of detecting the actual disturbance in the air of theexhaled breath of a person. This does imply that there are not otherstrong air currents in the cabin, room or vehicle (e.g., from an openwindow, a nearby air conditioning unit, a nearby heater etc.); if thereare, their effect on the measurement can either be discarded, or used todetect changes in airflow in the environment.

Large air currents will tend to be detectable as a low frequencymovement across range bins (i.e., a perturbation that flows across therange). This is more evident for sensing waveforms that have morereverberation (e.g., that allow the energy of one frequency to build upin the room, and associated room modes).

When considering a sleep staging system that works across a generalpopulation (i.e., including users with a normal healthy condition, userswith various health conditions, including respiratory conditions such assleep apnea, COPD, cardiac issues and so forth), it can be seen that thebaseline of respiration rate and heart rate can vary widely. Take forexample differences in age, gender, and body-mass index (BMI). Women mayhave a slightly higher baseline breathing rate than men for a similarage and BMI (although a recent study in children ages 4-16 does not showa statistical difference). Those with higher BMIs will tend to breathefaster than the average of somebody of a similar age. Children normallyhave much higher normal respiratory rate than adults.

Thus, in some versions, the system such as with processing device 100regardless of sensor type, may be made with a hybrid implementation,such as where initial signal processing and some hand crafted featuresare formed, prior to applying a deep belief network (DBN). (A hybridimplementation involves a mixture of human “hand crafted,” digitalsignal processing (DSP) derived features combined with features learnedby a machine. Initial supervised training is performed using expertscore polysomnography (PSG) overnight datasets from a sleep lab or homePSG, from multiple sites around the world, and scored by at least onescorer, using a specified scoring methodology. Further unsupervisedtraining is performed from datasets gathered with one or more of theselecting sensing methods. This allows the system to evolve to reflectnew and more diverse data outside of the sleep lab.

In terms of hand-crafted features (i.e., a human engineer/data scientisthas designed, chosen or created them), a breathing signal withassociated signal quality level is extracted, with specific features ofinterest being the variability of the breathing rate over differenttimescales, and the variation in inspiration and expiration time. Anestimate of a personalized baseline breathing rate for awake and asleepis formed. It is known for example that short-term changes in breathingrate variability while awake can be related to mood, and changes inmood, whereas these changes while asleep are related to changes in sleepstage. For example, respiration rate variability increases in REM sleep.Longer term changes in breathing rate itself can be related to changesin mental condition, such as providing indicators of mental health.These effects may be more profound when the user is asleep, especiallywhen analyzed over longer timescales, and compared to populationnormative values.

One can use the variability of the measured respiratory rate as anindication of the user's state (sleep/awake) or sleep stage (REM, N1,then N2, then lowest in SWS sleep). For example, when looking atnormalized respiratory rate variability over a period such as 15 mins ina normal healthy person, it is possible to see greatest variability whenthey are awake; this variability drops in all sleep states, with thenext largest being in REM sleep (but still less than wake), thenreducing further in N1, then N2, then lowest in SWS sleep. As an aside,air pressure due to breathing can increase in REM sleep, which can havean impact on the acoustic signal detected—a potential extra feature thatcould be detected in quiet environments or at quieter times.

Such normalized respiratory rate values should not vary significantlybetween different positions (supine, prone, on side etc.) for a healthyperson. However, it should be noted that calibration to the correcttidal volume is likely to be desirable. For example, the system maynormalize over the entire night since one person's average breathingrate might be, for example 13.2 breaths per minute (BR/MIN) while asleepwhereas another person's average might be 17.5 BR/MIN. Both ratesexhibit similar variability per sleep stage. The difference in rate ismerely masking the changes that may be considered for classifying thesleep states. The system can consider the average rate (or overall rategraph) for other purposes such as comparing to themselves over time, orindeed to someone in a similar demographic. For a person withobstructive sleep apnea (OSA), it is expected that respiratoryvariability will increase in the supine position (lying on back)—apotentially useful indication of the user's respiratory health.

Subjects with mixed apnea or central apnea tend to display largerrespiratory variability during wake than normal subjects (a usefulbiomarker), which those with obstructive apnea also have changes versusnormal during wake, which are not as obvious (but still present in manycases).

Person specific sleep patterns (e.g., breathing variability) can belearned by the system over time; thus, a system that can performunsupervised learning, once deployed in the field, is highly desirable.

These patterns can vary overnight (i.e., during a sleeping session) andcan be impacted by apneas occurring during the sleeping time, as partialor complete cessation of breathing (or paradoxical movement of the chestand abdomen when there is an obstructed airway). It can be seen that oneway to deal with this issue is by suppressing the periods with detectedapneas (and the associated oscillations in breathing rate), ifcalculating sleep stages. One can simply flag apneas and potentialmicro-arousals, rather than attempting to classify the sleep stage atthat point in time. Periodic breathing patterns, such as Cheyne Stokesrespiration (CSR), have a strong oscillatory pattern; these may also bedetected during a sleep pre-processing stage. While CSR can occur in anystage of sleep, the pauses tend be more regular in Non-REM sleep, andmore irregular in REM sleep (information which the system can use torefine sleep staging in subjects with CSR).

Similarly, a cardiac signal can be extracted with processing steps thatsuppress any harmonics relating to the breathing waveform morphology.Specific patterns such as obstructive, mixed or central apneas aredetected, along with any related recovery breaths, and movements relatedto gasping. From the cardiac signal, a beat to beat “heart ratevariability” (HRV) signal is estimated based on physiologicallyplausible heart rate values. Spectral HRV metrics can be calculated,such as the log power of the mean respiratory frequency, LF/HF (lowfrequency to high frequency) ratio, log of the normalized HF and soforth.

The HF spectrum of the beat to beat time (HRV waveform) is the power inthe range 0.15-0.4 Hz, relating to rhythms of parasympathetic or vagalactivity (respiratory sinus arrhythmia—or RSA) of 2.5 to 7 seconds, andis sometimes referred to as the “respiratory band”.

The LF band is 0.04-0.15 Hz, which is believed to reflect baroreceptoractivity while at rest (and some research suggests may have arelationship with cardiac sympathetic innervation).

The VLF (very low frequency) HRV power is between 0.0033-0.04 Hz (300 to25 seconds), and reduced values are related to arrhythmias andpost-traumatic stress disorder (PTSD).

HRV parameters can also be extracted using time domain methods, such asSDNN (standard deviation of normal inter-beat interval—to capture longerterm variability) and RMSSD (root mean square of successive heartbeatinterval differences—to capture short term variability). RMSSD can alsobe used to screen for irregularly irregular beat to beat behavior, suchas seen in atrial fibrillation.

In terms of HRV, a shift in the LF/HF ratio as calculated is detectablecharacteristic of Non-REM sleep, with a shift to “sympathetic” HFdominance during REM sleep (which may be related from sympathetic toparasympathetic balance).

More generally, there is typically increased HRV in REM sleep.

The longer term mean or median of the breathing rate and heart ratesignals are important for a specific person when analyzing overtime—especially if there is some intervention, such as a medication,treatment, recovery from an illness (either physical or mental), changein fitness level, change in sleep habits over time. They are somewhatless useful for comparing directly from person to person (unless to avery similar grouping). Thus, for breathing and cardiac variabilityfeatures, it is useful to normalize these (e.g., de-mean, remove themedian etc. as appropriate for the metric) such that that can bettergeneralize across a population.

Further analysis of extracted features can make use of a deep beliefnetwork (DBN). Such a network is composed of building blocks ofRestricted Boltzmann Machines (RBM), Autoencoders, and/or perceptrons. ADBN is particularly useful to learn from these extracted features. DBNscan be used without supervision, and then later trained with labeleddata (that is, data confirmed by a human expert input).

Exemplar human crafted “learn by example” extracted features that can bepassed onto the DBN, can include: apnea type and location, respiratoryrate and variability of same over different timescales, respiration,inspiration and expirations times, depth of inspiration and expiration,cardiac rate and variability of same over different time scales,ballistocardiogram beat shape/morphology movement and activity typessuch as gross movement, PLM/RLS, signal quality (integrity of measuresover time), user information such as age, height, weight, sex, healthconditions, occupation etc.). Other statistical parameters such asskewness, kurtosis, entropy of the signals can also be calculated. A DBNwill determine several features itself (“learns” them). Sometimes it canbe difficult to understand what exactly they represent, but they canoften do a better job than humans. A challenge is they can sometimes endup at bad local optima. Once they have “learned” the features, thesystem can tune them with some labelled data (e.g., data input by ahuman expert may score a feature (one expert or a consensus of severalexperts)).

The DBN can also directly learn new features from the input parametersincluding from the respiratory waveform, activity levels, cardiacwaveform, raw audio samples (in the case of SONAR), I/Q biomotion data(in the case of SONAR or RADAR), intensity and color levels (e.g., frominfra-red camera data) and so forth.

A machine learning approach that purely uses hand crafted features is a“shallow learning” approach that tends to plateau in terms of aperformance level. In contrast, a “deep learning” approach can continueto improve as the size of data increases. The approach discussed aboveuses deep learning (in this case a DBN) to create new features forclassic machine learning (e.g., take new features, a feature selectionwinnowing by feature performance, whiten with ICA (independent componentanalysis) or PCA (principal component analysis) (i.e., a dimensionalityreduction), and classify using a decision tree based approach such asrandom forests or support vector machines (SVM)).

A full deep learning approach, as used here, avoids such a featureselection step, which can be seen to be an advantage as it means thatthe system does not use sight of the huge variety seen in a humanpopulation. New features can then be learned from unlabeled data.

One approach for these multimodal signals, is to train a deep beliefnetwork on each signal first, and then train on the concatenated data.The rationale for this is that certain data-streams may simply not bevalid for periods of time (e.g., the cardiac signal quality is below ausable threshold, but there is a good quality respiratory, movement, andaudio features signal available—in which case, any learned or derivedfeatures from the cardiac data would be nonsensical for this period).

For classification, a sequence based approach such as Hidden MarkovModels (HMM) can be applied. Such a HMM can still optionally be used atthe output in order to separate the sleep stages, in order to map anoutput sleep graph to a stepped “sleep architecture” as might beprovided via a hospital sleep lab PSG system, and minimize unusual sleepstage switching. However, if we recognize that sleep is a gradualphysiological process, we may prefer to not force the system to a smallnumber of sleep stages, and allow it to capture gradual changes (i.e.,to have many more “in between” sleep states).

A simpler state machine approach with no hidden layers is possible, butultimately can have problems generalizing across a large population ofsleepers, each having their own unique human physiologicalcharacteristics and behaviors. Other approaches as Conditional RandomFields (CRF) or variants such as Hidden State CRF, Latent Dynamic CRF,or Conditional Neural Fields (CNF) or Latent Dynamic CNF. It should benoted that Long Short-Term Memory (LSTM) can have good discriminativeability, particularly when applied to sequence pattern recognition (moretypical in normal healthy sleepers).

Semi-supervised learning could be performed using a recurrent neuralnetwork (RNN), which can be effective in finding structure in unlabeleddata. An RNN is standard neural net structure, with Input, HiddenLayers, and Output. It has sequenced input/output (i.e., the next inputdepends on the previous output—i.e., hidden units have recurrentconnections that pass on information) using graph unrolling andparameter sharing techniques. LSTM RNNs are well known for naturallanguage processing applications (with LSTM to combat exploding andvanishing gradient problems).

In terms of detecting sleep onset, if a speech recognition service isrunning, voice commands by the user can be used as a second determinantof “wake” (not to be confused with nonsensical sleep talking). If apersonal smart device is used (unlocked by the user—then with UI input,movement of the accelerometer, gyroscope etc.), this can also be used asa determinant of wake to augment other sleep/wake sensing services.

Automotive Sensing (Related to Self-Propelled Vehicles or Machines):

Low frequency ultrasonic sensing can also be used in anautomotive/vehicular setting (e.g., in cars, trucks and other transporttypes). In-car entertainment (ICE), or in-vehicle infotainment (IVI), isa collection of hardware and software in automobiles that provides audioor video entertainment. The availability of associated speakers,microphones and processing electronics can be used for SONAR sensing.

There are several different types of applications.

For example, the technology can be implemented for security applicationssuch as to detect occupancy of a vehicle, such as for a child or babyaccidentally left in a vehicle, stowaways in the back of a truck, drivertaking a nap in a truck, or an intruder that has broken in to a vehicle.Thus, when a user leaves a vehicle the processing device may generate analarm if a person is detected by motion sensing within the vehicle, suchas upon sensing a door closure with a door sensor and sensing that thevehicle is turned off with a motor operations sensor. The sensingtechniques of the processing device 100 may also be configured to detecta person in a vehicle at an unauthorized time in a traditional intruderalarm situation and generate an alarm.

The technology of the processing device may also be implemented tomonitor breathing, cardiac signals and motion when a vehicle is stoppedor in motion to check health state, monitor fatigue, alertness, or sleepstate (asleep or awake) and/or sleep stages (light, deep, REM, stages 1to 3 etc.).

For vehicles with systems such as lane departure, seat sensor, and/oreye tracking, the technology can provide extra information about theattention level of the driver. For example, the processing device maydetect sleepiness by evaluation of sensed motion (e.g., respiratory,cardiac and/or gross body motion) and generate an alarm upon detectionof sleepiness.

For semi-autonomous vehicles, the processing device 100 can evaluatesignals/data from other sensors such as steering wheel pressure or usergrip (e.g., strain gauge, touch sensors or force sensors) and steeringwheel angle (e.g., optical sensors) in order to ensure that the driver'salertness level and health condition are such that the person remainsattentive for intervening if the driving system encounters an unusual ordangerous situation which would otherwise benefit from a humaninvolvement (e.g., requiring an ethical decision to be made).

For fully autonomous vehicles where user (driver) intervention is notrequired, the processing device 100 can be used to, as described above,so as to monitor user sleep and provide customized sleep programs toallow the occupant(s) to get a good sleep or nap so that the user canwake up, alert and refreshed at their destination. Optionally, theprocessing device can monitor the user's health condition and in theevent of detection of a dangerous condition of a user/passenger (e.g.,heart or respiratory failure) the processing device may generate acommunication with an alert (e.g., to a medical destination) and/orchange a navigation route of the vehicle to the medical destination(e.g., nearest hospital) and set/control movement of the vehicle with anautonomous control system to drive to the medical destination.Optionally, in response to such a detection, a semi-autonomous orautonomous control system may control the vehicle to come to a safe stopand generate a communication (e.g., an automated voice mobile telephonecall) identifying location of the vehicle to emergency medical supportand identifying the nature of the emergency.

The technology can be implemented: 1/ in existing vehicles withspeaker/mics and vehicle control systems that can be upgraded (e.g., byupdating the system's software), 2/ in new vehicles infotainmentdesigns, add-on/aftermarket systems can be installed to enable thesensing and associated services, and 3/ in any vehicle, portable systemssuch as smart phones or smart speakers can be activated and used for thesensing and user interface.

Support Existing Fleet of Regular Vehicles—Security and Protection

For example, the sensing capabilities can be implemented in a car usingan existing stereo system (for example where separate tweeters(loudspeakers designed to reproduce high frequencies)) are included,although simple single cone full range speakers can also be effective)to detect absence and presence. Preferentially, this sensing capabilitywould operate when the car is running, but it could also be employed atvery low power levels when the vehicle is parked to detect a person oranimal breathing for safety reasons (e.g., to detect and raise an alertif a child or baby or indeed a pet was left in the car accidentally). Anadvantage of such a system is that it does not require changing ofexisting car hardware. For example, the sensing capability can beimplemented as a software upgrade to a head-unit/amplifier (especiallywhere the head-unit supports applications). Thus, existing car fleetscan be upgraded at relatively low cost. In many such cases, such systemsinclude one or more microphones having noise cancellation features andare of sufficient quality to serve as sensors as described herein. Inthis regard, they have a frequency response capable of sensing the audiosignals described herein (e.g., low ultrasonic frequencies).

Add-on Kits/Smartphones

Aftermarket Bluetooth car kits (either wired in to the car speakers orstandalone) may also be upgraded to include the sensing technologiesdescribe herein.

Furthermore, if a smartphone is used in the car, the smart phone couldoperate as a processing device including an application that provides aconfiguration for in-vehicle sensing.

Speaker and Mic Locations for Sensing

Careful selection of speaker and microphone location is beneficial forachieving accurate sensing of the driver, such as where hand movementsrelated to steering wheel, gear change, and leg movements are expected.Use of dash locations near the A-pillar and door card or door panel(which are actually quite common in multi speaker systems) can provide agood view of the chest. As a result, typically, passengers will haveless “motion” than the driver.

If more speakers than microphones are available, the processing device100 may operate different individual speakers to monitor differentindividuals, while sensing with a common microphone. For example, theprocessing device may generate sensing signals at different sensingfrequency ranges for different speakers. If possible, separatemicrophones may be configured in the front and rear of the car toprovide some spatial separation. Ideally, a speaker and microphonecombination may be implemented for each seat location (i.e., for eachperson), as part of the overall in car entertainment system.

There are a variety of different sensing scenarios (which can benefitfrom different sensing waveforms/schemes) such as the scenario to detectone or more specific person(s) in a car (e.g., individual sensing foreach seat) versus detecting anything in car (which may be more useful asan occupancy detector for security/safety purposes) versus detectinganything in a larger space (e.g., a large truck, shipping containeretc.).

One challenge relates to differences in noise level with engine noise(internal combustion engine), hybrid and electric powertrain cars. Theremay be engine/motor noise, mount transfer, body transfer, and resultinginterior noise from same. Other than powertrain induced noise, there aredifferences in tire noise ratings (as well as wear levels), rattling ofthe car body, size and shape of the cabin (e.g., dead space and echoes),wind induced and road induced noise. A car is a difficult place tocritically listen to a sound system because of noise and the cabinconfiguration. Reflections, standing waves, resonations, uneven interiorsurfaces, resonant frequencies, and less-than-adequate space for properspeaker placement can affect sound quality.

In terms of directionality of speakers, sensing can be affected by wherea speaker is pointed at a person (e.g., if placed in the door card) orwhether it requires multipath detection (e.g., a dashboard speakerreflecting from the windscreen, a rear speaker facing upwards in therear parcel shelf).

There is scope for significant reverberation in car (as it could crudelybe considered as basically an enclosure), but there can be a benefit inthat the user is generally in a relatively static location.

Cars are typically designed as a means of transportation first, and as alistening environment much later (potentially one of the much laterdesign steps). This may change over time as infotainment systems areimproved.

Ray tracing is a predominant means of modelling car acoustics at >4 kHz.This is applicable for a new car design, but may not be feasible for aretrofit system, where the system seeks to quickly model/map theinterior of the car (e.g., with an audible or inaudible probingsequence). Changes in furnishings, seat types, dashboard reflections,differences in windows down (partially or completely) affect sounddifferences from one vehicle to the next. There can be significantdifferences between an SUV and other types of cars (e.g., a cabrio/softtop/targa configuration).

The most straightforward vehicle configuration has tweeters (speaker)pointed towards the front seats (which is a common setup used by carmanufacturers). In this case, there may be a specified distance fromspeaker to midpoint of a seat, and the system can model the depth of aperson sitting on the seat. The system can detect the side and front ofthe person for the case of a speaker mounted high up in the door (atypical location for a tweeter) or on the side of the dash or A-pillar.Thus, the frequency range and modulation scheme may be tailored to thelikely distance by the system. The system can also select a scheme thatis robust to reverberation noise (i.e., that does not allow the sensingenergy at any one frequency to build up in the car). One such example isa scheme employing dual ramped FMCW. Such sensing may be turned on andoff by the system between each cosine-like sequence, and also the systemmay adjust the ramped frequencies between each sequence (takingadvantage of the “zero” points to avoid audible clicking of thespeaker). Such system adjusting of frequencies is desirable in anautomotive setting, due to the variable nature of in-cabin noise. Forexample, an internal combustion engine based sports car with a sportsexhaust system and significant induction noise is very different to anelectric car, although the electric power train can have whine at higherfrequencies.

Where premium audio systems are installed, or aftermarket systems, thetweeters can be rated to over 30 kHz, which provides a wide operatingband (assuming that the amplifier components are using an adequatesampling rate to avoid aliasing). Depending on the tweeter manufacturer,an upper range of 21-30 kHz may be available. A system may use slightlyhigher or lower frequencies depending on manufacturing variance ofspecific parts. Availability of such frequencies may be detected by asetup process (i.e., to determine the capabilities of the partsinstalled).

Microphones are often placed in the headliner for active audiocancellation or hands-free phone use. These are in a useful position todetect the reflected signals from the front passengers. Proposed audiocancellation systems typically have speakers mounted midway up the door,and multiple microphones (e.g., at least 4 mics in the vehicle). Thistype of configuration is suitable for biometric detection and biometricidentification/recognition. Microphone frequency response willultimately limit the highest usable frequency (e.g., a mic rolling offat 20 kHz with tweeters capable of 25 kHz will ultimately limit thesystem to 20 kHz). Availability of such frequencies may be detected by asetup process (i.e., to determine the capabilities of the partsinstalled).

Other configurations may be implemented without standalone tweeters,where there are larger cone speakers lower down in the doors that canstill produce above 18 kHz. This means that lower frequency rangesshould be used.

There may be only one microphone placed near the driver. In this case,the system can still function, as different frequency bands and/orcoding schemes (e.g., re-using frequencies ranges, but separated intime) can be used. So four passengers could utilize four speakers whereeach speaker monitors one person. This may be implemented, with ideallyfour microphones. However, it could be implemented with one microphone,where the processing device select different range bins in an FMCWsystem (i.e., use the estimated time for the sounds to reach the torsoof the respective passenger) for the different passengers.

Heating, Air Conditioning, and Windows

The processing device 100 may be configured with the vehicle sensingsystem so that it can detect whether the car windows are open versusclosed, such as by accessing a signal from a window sensor. In someversions, the processing device may be configured to communicate with aCANBUS system of the vehicle (where available) to receive signals fromany of the sensors of the vehicle. For example, to automatically adjust(or query the current position of) electric windows, a message can begenerated by the processing device on the vehicle network that iscommunicating with the appropriate door module (e.g., a passenger doormodule). Some of these implementations are vehicle manufacturer systemspecific (i.e., not part of the OBD protocol), and can use data from anexisting position sensor.

For example, different parameters of air conditioning/fan settings(e.g., detecting which vents/flaps are open (direction of airflow), fanspeed etc.) on air blower system can also be read from the vehiclecontrol systems by the processing device, and serve as a basis formaking adjustments to the sensing signal (e.g., changing its amplitudeand/or waveform(s)) to facilitate suitable physiological parametersignal quality recovery with the sensing signal. Such adjustment couldinclude adaptively changing the frequency band that is applied forsensing to one that is suffering less interference, and/or to change thesensing waveform to have more or less reverberation. Other parameterssuch as detecting if the car/vehicle doors open or closed, or steeringwheel hand removal etc. can also be obtained by the car sensors.

Autonomous Vehicles—Sleep, Fatigue and Alertness Sensing

Artificial intelligence systems for autonomous driving are on the rise.Depending on the autonomy level of the autonomous vehicle, the desiremay be to promote sleep on the move or stopped, or to detectfatigue/sleepiness prior to sleep to make an intervention.

In fully autonomous vehicles, a steering wheel, pedals and othercontrols might not be included. This provides new opportunities in termsof re-using the vehicle compartment space as sleeping quarters or a workarea (or even for exercise) during travel/commuting.

The persons in the vehicle can have their vital signs monitored bothwhile awake and asleep using, for example, the low frequency ultrasonicsensing.

When they are asleep, the processing device 100 can have theirrespiratory waveforms checked in order to screen for sleep apnea, andbreathing and cardiac rates checked for signs of illness. The processingdevice 100 can then present a sleep score, as well as a fatigue score.The system offers the opportunity to manage insomnia, and other sleepconditions.

Breathing entrainment exercises can be provided to help induce sleep insuch an autonomous vehicle. The whole environment of the vehicle canautomatically adjust to promote good quality sleep. For example, theprocessing device 100 serving as an infotainment system can beconfigured to produce active noise cancellation to reduce vehicle noise.In some cases, the processing device might control light within thevehicle. For example, it might make a controlled adjustment to windowtransparency to reduce light or raise light for wake time such as withcontrollable glass or electrochromic glass to darken or lighten). It maycontrol automatic blinds on the windows for light adjustments and/orother devices to set acoustic treatments or sound barrier (e.g., soundabsorbing window covers) in positions to make a quiet environment in thevehicle.

When they are coming close to the destination, the processing device ofthe system, can consider, for example, the GPS vehicle location from aGPS sensor of the system, time of sleep or travel, and/or the detectedsleep state of the users, can control these devices to increase lightand sound levels in order to gently wake the user.

As a passenger in a “robo-taxi” or ride sharing vehicle, the processingdevice can include a relax and sleep program to induce either with soundand/or visualizations controlled and produced by the processing device.Optionally, the processing device may also be configured to evaluaterespiration during sleep such as to carry out a sleep apnea test on themove (e.g., counting arousals, apneas, hypopneas) and reporting them tothe user on waking.

Notably, the above applications/operations may also be implemented insystems (e.g., entertainment systems) included within standarddriver-operated busses and trains currently providing transportation onroads and railways. These vehicles are often used by passengers for longdays and/or overnight trips. These operations may also be implemented inentertainment systems of airplanes.

In a semi-autonomous vehicle, the detection of wake (rather than sleepor pre-sleep) is important to ensure the driver can react to anyunanticipated road situation as part of the human-to-machine interface.For example, depending on the level of autonomy of the autonomousvehicle, a processing device of such a vehicle may require the user tobe able to intervene—i.e., they must be alert and “present”. Such asystem could be integrated to communicate with other car drowsinessaids, such as sensors in seats, cameras performing eye tracking etc. Thesystem may implement a voice assistant to talk to the driver to judgetheir current alertness level based on the responses, and take action ifthey are likely to enter micro-sleep. For example, such a car mayincrease sound as an alert and/or pull the car off of the road and stopto allow the driver to sleep or wake up. Thus, motion detectioncapabilities such as the sleep or wake detection operations of theprocessing device 100 may serve as an input to vehicle controloperations so as to control an adjustment of a vehicle operation (e.g.,movement of the vehicle, reduce speed, stop, pull over, and/or changedestination or offer a navigation destination to a location to helpdriver (e.g., coffee shop or hotel)) based on the detection of wake orsleep.

In some cases, based on such detections, the system may controloperations to discourage sleep. For example, it may generate a sound,such as a distracting sound. Such sound may help for a short time beforethe person falls asleep—but may be enough to allow them to drivesomewhere safe to stop for a coffee or nap—or to activate a fullautonomous return-to-home navigation and/or autonomous driving function.In example, a distracting sound might include starting with a defaultcabin chime or other vehicle error sound. Such a sound may then includespoken commands to wake up. Operations may include shaking the drivingwheel (such as with integrated vibrator(s)) or even the car seat back(such as with integrated vibrator(s)) by control of a mechanical shakingdevice.

User Personalization within and Between Vehicles

There is a move from car ownership to a “mobility as a service” businessmodel. This implies that users no longer need to own a car. Longerlasting cars also promote such a service (i.e., move from internalcombustion engines to electric motors) as well as urban/city living, andimprovements in car and ride sharing services. It may also be promotedby requirements of city authorities to reduce congestion and increaseair quality.

Processing devices of such vehicles can be configured to allow transientpersonalization of the vehicle with user-customizable parameters. Thisallows personalizing the user experience across many vehicles—whether itis in a car that is regularly re-owned, a car sharing service, or a ridesharing/public service vehicle. Such parameters can include a customizedand personalized sleep/nap configuration—which is particularly relevantwhere there is a separate human driver, or an autonomous vehicle.

For example, the processing device 100 of the vehicle may monitor theuser's sleep in challenging automotive environment—including for examplenoise, motion, fans/airflow. In order to promote sleep, the processingdevice may be configured to select and optimize a navigation route,adjust suspension settings and/or automated vehicle control style (e.g.,driving casual rather than aggressive such as by adjustments toacceleration and/or braking) to enhance the user's sleep. Subject touser's instruction or it current operational mode, the processing device100 of the vehicle may even select a longer route to a destination toallow sufficient time for a nap (in order that the user(s) gets part orall of a sleep cycle in order to wake up refreshed/minimize sleepinertia), and/or for comfort (select a route with better road surfaces,fewer predicted braking/acceleration or cornering events, etc.).

The processing device 100 may control a light adjustment and windowcoverage adjustment so that the vehicle changes the interior lighting(both intensity and hue). The processing device 100 of the vehicle maythen play music and/or apply active noise cancellation or masking sounds(e.g., white noise) such that the environment is optimized forrelaxation and sleep. For a vehicle with seats that can adjust or a bed,in the relax phase, the processing device 100 may control a massagebased on controlling motors and actuators in the seats. The processingdevice 100 may then mute non-safety-critical notifications during thesleep or nap time. For a longer trip, the system can offer any one ormore of the following programs: relax, sleep, waking, work program. Theprocessing device 100 may control a switch to work, relax, sleep,waking, work programs to suit (or be more aligned to) the circadianrhythm of the user. For example, the processing device might offer a nap(or delayed a requested one) to correspond to an afternoon “dip” of auser, where the user is on a long journey.

Pre-Sleep in Moving or Stationary Vehicles

For people prone to motion sickness, a nap or sleep time with an initialrelax phase using deep breathing (with optional audio cues orsynchronized music provided by the processing device 100 to guide theuser to achieve deep breathing) with eyes closed, can help reduce anysuch sickness. The vehicle suspension may be controlled by theprocessing device to perform more correcting actions in order to reducethe feeling of motion sickness, and may move a sleeping position (e.g.,a controllably adjustable seat) towards the front of the vehicle if manycorners and bumps or grade changes are expected on the route.

The processing device 100 could start a guided breathing exercise withmonitoring the user's breathing rate to estimate an initial baselinerespiration rate, and baseline inspiration time to expiration timeratio, along with depth of breathing (e.g., shallow inspiration andexpiration). It can then provide an audible or visual (but audible ispreferred if the user is relaxing and intending to sleep) cue for theperson to adapt to inhale for a count of four seconds, and then exhalefor a count of four seconds, all through the nostrils (i.e., mouthclosed). If the user can sustain this for 10 or more breaths, they maybe allowed a short recovery time, then guided to slower breathing(moving from 4 to 6 to 8 seconds for inhalation, and the same forexhalation) in order to determine what is comfortable, achievable, andsustainable (based on monitoring the actual respiration waveform). Aguideline time is 5 to 15 minutes for this exercise. In terms ofpositioning in the vehicle, this depends on whether the breathingprogram is for relaxation, or sleep—and the adjustment of the seat/bed.

Optionally, the processing device may generate control signals tocontrol a seat that include motors/actuators so that the seat acts as amassage seat.

This can be extended to an abdominal (diaphragm) breathing exercise. Ifthe person is in a seat that is relatively upright, they may be asked toplace one hand on their chest, and another hand on their lower stomach.They are prompted to breathe deeply via their nostrils such that theirlower hand moves, but not upper hand (as much as possible). Gesturerecognition sensing can be performed by the processing device 100 usingits sensing operations to confirm that the lower arm is moving more thanthe upper arm. Heart rate and detection can be performed to confirm thatthe exercise is having the desired effect (reducing heart rate, andincreasing heart rate variability to their personal baseline “relaxed”ranges).

As the user falls asleep, the seat can be controlled by the processingdevice to adjust it to provide suitable support.

For longer journeys, the system can adjust navigation parameters to addcomfort breaks to the sleep/nap program, to bring the user to a safeplace with a restroom and/or an area for a walk. This might becoordinated with a battery or fuel recharge, or battery swap event.

Sleep Disordered Breathing (SDB) Screening

Where non-contact sensing is deployed by a processing device in a publicservice or ride sharing vehicle, the passenger(s)/user(s) (i.e., singleor multiple occupancy) can be asked by a voice assistant of the deviceif they wish to opt in to physiological monitoring. If they choose to doso, while awake, the system can monitor their cardiac and respiratoryparameters automatically. If they sleep with one or more sleep cycles,the system can monitor their sleep stages, as well as check for apnea orhypopnea events. Such a system can be implemented in any vehicle such asa non-autonomous car where there is a driver, in an autonomous car aspart of a car sharing service or taxi service, or a vehicle with aberth/sleeping area (truck, train, plane, etc.).

System for Alertness

Non-contact sensing of alertness can be used to augment safety systemsin vehicles where the user may be requested to make a safetydecision—such as overriding an autonomous vehicle, or as part of a lanedeparture detection or drowsiness detection system. Such sleepiness orfatigue detections may be made using any of the methodologies describedin International Patent Application No. PCT/AU2014/059311.

Security Applications

Illegal immigration can be an issue for customs and immigration as wellas prison authorities, police, and freight operators.

It can be seen that non-contact sensing by a vehicular processing device100 may be configured to detect the presence of a respiratory signal ina monitored space (such as when the vehicle is parked) and alert thedriver and/or a central monitoring service. The system may be configuredto detect masking and/or interfering signals that affect the sensingsignal so that changes may be made to the sensing signal to avoid theinterference. Thus, the processing device of the system may correctlydetect a respiratory signal, or otherwise report a sensing fault ifinterference is detected. The system can benefit from the fact ofmultipath modal behavior to perform biometric identification of aperson, with detection of unique biometric signatures. Such signaturescan be influenced by the cabin (e.g., the system can select sensingsignals that cause reverberation in the sensing space—akin to detectingripples on the surface of a swimming pool).

Protect Vehicle

Vital signs monitoring in the vehicle can be used to confirm anintrusion into the vehicle, and work with other intrusion sensors (e.g.,door contact switches and car alarm systems).

Protect Persons (Children or Babies) or Pets Left Unattended in Vehicle

As previously described, by detecting respiration and heart rate in arecently vacated or intended vacant vehicle, alerts can be generated bythe processing device if a child or baby or pet is left accidentally inthe vehicle when unattended. The vehicular processing device may thengenerate one or more a vehicle control signal(s) to automaticallyimmobilize the vehicle (to prevent stealing) or engage systems such asventilation, cooling or heating in order to protect the occupant untilhelp arrives.

Automatic Feedback to the User Via a Virtual Help Agent

Unlike simple scripted or tree-based questions/answers approach, theprocessing device may be implemented with a fully personalized virtualagent application in order to process multimodal signals (includingnatural language processing—such as for speech recognition) using deeplearning approaches for advice delivery.

One issue that arises, especially in relation to saving the user's dataand any interaction of the system with the user, such as with a sharedvehicle system, is the security of the saved data. This can be addressedif at least some of this data, such as the delivery of advice to theuser (which can include medical advice) is enabled via blockchain inplace of a classic database. The blockchain works on the basis ofdistributed network consensus with cystography, to make digital events(such as delivering advice, and the data to inform this advice)immutable and very hack resistant. For example, this can ensureintegrity between disparate internet of things (IoT) sensors—particularly to enable interoperability with trust and sharedaccountability. It can allow a decentralized health exchange to reduceadministration of the system required. The user can be assured of theprivacy of the interaction—for example with a medical doctor.

In terms of the sharing of anonymous data for research purposes,blockchain can identify management features with pre-defined user accessthat can allow access to medical (and/or protected health information)and immutable storage of encrypted data.

It can be seen that several types of blockchain could deliver thesebenefits —including public, consensus, and private blockchain.

A public blockchain is by definition available to everybody, based onthe tenets of cryptoeconomics, such as proof of work and proof of stake.Consensus and private blockchain mean that user access is restricted,but maintain some or much of the partial guarantees of authenticity thatblockchains provide, with some or little decentralization. A consensus(basically a hybrid) blockchain could allow a group of health andwellness companies along with certain medical providers to collaborateto deliver a medical advice service, connecting IoT and other sensingdevices, advice engines based on artificial intelligence (AI) andmachine learning (ML), and smart contracts (payments on the sameblockchain) between the user and physician to deliver targeted care.

Power Nap

The processing device 100 of the system can be programmed with a napfunction to assist with a smart nap, whether the person is lying in bedor seat of a vehicle. For example, the user can tell the device “I amtaking a nap.” The interactive audio device can then audibly help theuser fall asleep and then, by monitoring sleep and/or time, guide aperson to an appropriate duration of nap based on the expected availabletime, an estimate of current sleep deficit, time of day, and userrequest. For example, it may optimize to target durations such as a 20min, 30 min, 60 min or 90 min (full sleep cycle). The 60 min nap is tooptimize deep sleep and allowing some time to recover at the end fromany sleep inertia, whereas the 20 and 30 min target times are optimizedto wake the user while they are still in light sleep, or before being indeep sleep for more than a minute or two. The time awake before sleep(nap) is also recorded in addition to the nap time.

A 20-25 min nap can be preferential to a 90 min full sleep cycle if theuser is sleeping normally in the preceding days, as the longer periodcan impact sleep that night.

In an autonomous vehicle with or serving as processing device 100 of thesystem, the user could ask the device for a daytime nap (say at lunch).The vehicle can arrive to collect them, lull them to a nap by eitherdriving quietly for the duration, or finding a safe place to park. Thiscould be before or after the person has eaten, with a preference for thenap after lunch to allow time to digest (but may need to allow a littletime such as 10-20 mins to reduce any possible acid reflux).

Example Vehicle Processing Applications

An example control processing methodology of the processing device 100may be considered in reference to FIG. 8. At 802, the processing devicemay sense or detect presence in a vehicle cabin such as with the motionsensing methods previously described. At 804 a number of vehicleoccupants may be determined. At 806, for the determined vehicleoccupants, the processing device 806 may attempt to identify eachoccupant such as using biometric characteristics derived from motionsensing techniques as previously described. At 810, the processingdevice may access data or signals from other vehicular sensors (e.g.,door closure status sensor, seat belt status sensors, wireless key/keyfob detected etc.). Based on the sensors and/or motion sensing, at 812the processing device determines that the vehicle is or had been lockedstate, such that no occupants were expected. At 814, the processingdevice detects, based on sensed physiological signals (e.g., respirationand/or cardiac signal in ranges attributable to children or infants),that only a child and/or infant occupies the vehicle. Alternatively, at816, an occupant is detected but not recognized such that processingdevice may determine an unauthorized presence (intruder or stowaway).Based on the determinations at 816 or 814, the processing devicegenerates output such as to trigger an alert (e.g., alarm orcommunication). At 820, the processing device may also generate output,based on the prior determinations, such as one or more control signalsto a vehicle control system to disable the vehicle in the event of anintruder or stowaway. Optionally, at 820 in the event of a detectedchild/infant left behind, the generated output to a vehicle controlsystem may be a control signal to activate a vehicle environmentalcontrol (e.g., ventilation and/or temperature control), for example.

At 822, the processing device determines, such as by key FOB detectionand/or biometric identification, that the vehicle is in use/occupied byan authorized person. At 824, conducts a deep learning cardiorespiratoryanalysis, such as by the methods previously described. The processingdevice may then operate sleep analysis/detection processes at 824 aspreviously discussed, fatigue/alertness detection processes at 828and/or medical screening/service processes at 830 such as using themotion sensing techniques described herein. At 832, the processingdevice may optionally generate output based on the detection processesat 826, 828 and 830 such as by engaging the user with audio/visualinteractive communications (e.g., AI process and/or chatbot) using, forexample, a speaker and microphone, coupled with the processing device.Additionally, at 834, the processing device may optionally generateoutput such as one or more control signals for setting or requestingoperations of a vehicle control system (e.g., movement of the vehicle,adjustment of a light condition of the cabin vicinity, adjustment ofelectrochromic glass transparency, movement of a seat of the cabinvicinity, adjustment of a braking parameter, adjustment of anacceleration parameter, adjustment of a suspension setting, adjustmentof window coverage, adjustment of an acoustic barrier, immobilization ofthe vehicle, engagement of vehicle ventilation and/or engagement ofvehicle cabin cooling/heating system) based on the interactivecommunications and/or the processes at 826, 828 and 830. Additionally,at 834 and/or 832, output may be generated to communicate data includingthe nature of the detections made at 826, 828, 830 and 832. Optionally,at 836, data concerning the detected conditions may be recorded, such asin a secure manner. For example, the data may be stored by a blockchainprocess at 836.

FIG. 9 illustrates additional example processes that may be implementedby a vehicular processing device of the present technology, such as foraudio based sensing. Optionally at 902, a vehicular processing device,such as an audio entertainment system in a housing (e.g., dashboard) ofa vehicle, may receive a download of a processing application withcontrol instructions for execution by one or more processors of theprocessing device. At 902, the processing device may check capabilitiesof the speaker/microphone of the system such as to confirm capability ofgenerating and receiving low frequency ultrasonic sensing signals. At904, setup processing may be run by the processing device to determineparameters for acoustic sensing. This may optionally involve generatingand sensing various acoustic sensing signals. For example, at 906 audioprobing sequences as previously described may be generated to map theinternal area (cabin) of the vehicle. Optionally at 908, pre-configuredvalues may be accessed based on known sensing characteristics (map) ofthe particular vehicle and sensing system.

Following these processes, the processing device may control generationand reception of sensing signals, such as by detecting one or morepersons within the vehicle cabin at 910. Such sensing may be based ondetection of motion and/or analysis of detected motion to detectphysiological characteristics (e.g., cardiac motion and/or respirationmotion). Such detection may serve to identify particular locations ofthe cabin that are occupied by a person. Upon confirmation of suchdetections, the processing device may activate one or more processes.For example, at 912, the processing device may detect ambient vehiclenoise or other sounds in the environment (e.g., music voices, etc.) suchas with a microphone. Such sounds may serve as information for filteringor adjusting of sensing signals as previously described. At 914, theprocessing device may determine, such as by access to vehicle controlsystem sensors or vehicle sensors, climate control system settings. Suchinformation may also serve as information for filtering or adjusting ofsensing signals as previously described or assist in evaluation ofphysiological movement signals for their characterization. At 916, theprocessing device may access other vehicle sensor information such asdetermining steering wheel angle, accelerator/brake pedal setting, seatbelt status, etc.). Such information may also serve as information forfiltering or adjusting of sensing signals as previously described orassist in evaluation of physiological movement signals for theircharacterization. At 918, other sensors, such as chassis level,throttle, suspension, drive train, motor sensors, may be accessed. Suchinformation may also serve as information for filtering or adjusting ofsensing signals as previously described or assist in evaluation ofphysiological movement signals for their characterization. At 920, theprocessing device may perform physiological sensing (e.g., with acousticsound generation SONAR) and characterization of motions that aredetected with such sensing, such as according to any of the detailspreviously described.

FIG. 10 illustrates additional example processes that may be implementedby a vehicular processing device of the present technology, such as forsensing in an semi-autonomous or autonomous vehicle. At 1002, thevehicular processing device detects entry of a user into the vehicle. At1004, such as in response to an interactive query (e.g., naturallanguage processing) by the processing device, the user is prompted toselect a monitoring or screening service. At 1006, while the user isstill awake, the processing device may begin sensing by the processesdescribed herein to detect awake physiological characteristics. Such aprocess may optionally include a sleep inducement presentation such aswith respiratory entrainment. At 1008, the processing device may performsleep analysis (e.g., stage detections) and otherwise detect sleep. Theprocessing device at 1008 may control a nap process (e.g., power nap) aspreviously described. At 1010, 1012 and 1014, the vehicular processingdevice may optionally perform health screening. For example, in anoptional sleep disordered breathing detection process 1014, events ofSDB may be detected via motion and/or sound sensing as previouslydescribed. Optionally, a respiratory screening detection process 1012,respiration may be monitored such as for detection of chronic diseaseconditions (e.g., worsening of heart failure) via motion and/or soundsensing as previously described. Optionally, in a cardiac screeningdetection process 1010, cardiac information may be monitored such as fordetection of chronic disease conditions (e.g., worsening of heartfailure) or other cardiac related occurrence via motion and/or soundsensing as previously described.

At 1016, the processing device may generate output based on thedetections of any of the screening process(es). As described in moredetail herein such output may be one or more vehicle related controlsignals and/or data indicating the detections of any of these processes.At 1018, data of the detections may be secured, such as with ablockchain recording process. At 1020, optional payment services mayarrange for currency transfer, such as in relation to a charge for thescreening, which may be based on the results and/or a transactionrecorded in the blockchain data. At 1022, the processing device maycommunicate such result with a health care provider or emergency healthservice. Optionally, at 1024, the processing device may communicationwith a medical institution such as a clinic or hospital for services.

FIG. 11 illustrates example processes for a sleep or nap service, thatmay provide guided relaxing breathing exercises, and optionally usemotors/servos/actuators in a seat to act as cues or tactile feedback. At1102, a personalized sleep service is activated, such as with voicecommand recognition and/or presence detection by the vehicularprocessing device. At 1104, the processing device may detect loadconfiguration and optionally perform biometric identification. At 1106,vehicle and sensor capabilities are determined, such as for meetingpre-existing a user parameters accessed based on the biometricidentification. At 1108 a nap/sleep plan may be initiated such as todetermine desired length of sleep time parameters. At 1110, a navigationsystem may compute a destination and route based on the sleep parametersas well as, for example, terrain information, distance information,sleep and travel time information, weather information and trafficinformation. Such a destination may optionally be provided by the user.Optionally, arrival at such a destination via the computed route may becontrolled by an autonomous vehicle operations control system. At 1112and 1114, cabin environment is controlled such as with a vehicleenvironment control system. As previously described, such control mayinclude temperature, air, humidity, and light adjustments. At 1116, asleep inducement or relaxation inducement presentation is made, such aswith generated audio, tactile (e.g., seat adjustment) and visual output,for example by entraining breathing. At 1118, the processing device, viaa vehicle operations control system may adjust a seat characteristic tomove the seat to a comfortable position and formation for the user. At1120, the processing device may further adapt light and sound in thevehicle, such as with noise cancellation and noise masking generation(e.g., white noise). At 1122, the user's sleep is tracked by theprocessing device, such as by sensing motion as described herein todetermine sleep, sleep time, sleep score, sleep stage, etc.) At 1124,the processing device, such as at a predetermined time, sleep stage, anddestination, may generated output to wake the user, such as withcontrolling generation of sound, seat motion and/or light adjustments inthe vehicle.

5.2 Other Remarks

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in Patent Office patent files orrecords, but otherwise reserves all copyright rights whatsoever.

Unless the context clearly dictates otherwise and where a range ofvalues is provided, it is understood that each intervening value, to thetenth of the unit of the lower limit, between the upper and lower limitof that range, and any other stated or intervening value in that statedrange is encompassed within the technology. The upper and lower limitsof these intervening ranges, which may be independently included in theintervening ranges, are also encompassed within the technology, subjectto any specifically excluded limit in the stated range. Where the statedrange includes one or both of the limits, ranges excluding either orboth of those included limits are also included in the technology.

Furthermore, where a value or values are stated herein as beingimplemented as part of the present technology, it is understood thatsuch values may be approximated, unless otherwise stated, and suchvalues may be utilized to any suitable significant digit to the extentthat a practical technical implementation may permit or require it.

Unless defined otherwise, all technical and scientific terms used hereinhave the same meaning as commonly understood by one of ordinary skill inthe art to which this technology belongs. Although any methods andmaterials similar or equivalent to those described herein can also beused in the practice or testing of the present technology, a limitednumber of the exemplary methods and materials are described herein.

When a particular material is identified as being used to construct acomponent, obvious alternative materials with similar properties may beused as a substitute. Furthermore, unless specified to the contrary, anyand all components herein described are understood to be capable ofbeing manufactured and, as such, may be manufactured together orseparately.

It must be noted that as used herein and in the appended claims, thesingular forms “a”, “an”, and “the” include their plural equivalents,unless the context clearly dictates otherwise.

All publications mentioned herein are incorporated herein by referencein their entirety to disclose and describe the methods and/or materialswhich are the subject of those publications. The publications discussedherein are provided solely for their disclosure prior to the filing dateof the present application. Nothing herein is to be construed as anadmission that the present technology is not entitled to antedate suchpublication by virtue of prior invention. Further, the dates ofpublication provided may be different from the actual publication dates,which may need to be independently confirmed.

The terms “comprises” and “comprising” should be interpreted asreferring to elements, components, or steps in a non-exclusive manner,indicating that the referenced elements, components, or steps may bepresent, or utilized, or combined with other elements, components, orsteps that are not expressly referenced.

The subject headings used in the detailed description are included onlyfor the ease of reference of the reader and should not be used to limitthe subject matter found throughout the disclosure or the claims. Thesubject headings should not be used in construing the scope of theclaims or the claim limitations.

Although the technology herein has been described with reference toparticular examples, it is to be understood that these examples aremerely illustrative of the principles and applications of thetechnology. In some instances, the terminology and symbols may implyspecific details that are not required to practice the technology. Forexample, although the terms “first” and “second” may be used, unlessotherwise specified, they are not intended to indicate any order but maybe utilized to distinguish between distinct elements. Furthermore,although process steps in the methodologies may be described orillustrated in an order, such an ordering is not required. Those skilledin the art will recognize that such ordering may be modified and/oraspects thereof may be conducted concurrently or even synchronously.

It is therefore to be understood that numerous modifications may be madeto the illustrative examples and that other arrangements may be devisedwithout departing from the spirit and scope of the technology.

1. A processor-readable medium, having stored thereonprocessor-executable instructions which, when executed by a processor ofan electronic device, cause the processor to process data sensed in acabin vicinity of a vehicle, to detect physiological movement of a user,the processor-executable instructions comprising: instructions tocontrol producing a sensing signal in the cabin vicinity of a vehicle;instructions to control sensing, with a sensor, a reflected signal fromthe cabin vicinity of the vehicle; instructions to derive aphysiological movement signal with at least a portion of the sensedreflected signal and a signal representative of at least a portion ofthe sensing signal; and instructions to generate an output based on anevaluation of at least a portion of the derived physiological movementsignal.
 2. The processor-readable medium of claim 1 wherein the sensingsignal is any one or more of a radio frequency sensing signal generatedby a radio frequency transmitter coupled with the electronic device, anacoustic sensing signal generated by a speaker coupled with theelectronic device, and an infrared sensing signal generated by aninfrared emitter coupled with the electronic device.
 3. Theprocessor-readable medium of any one of claims 1 to 2 wherein the signalrepresentative of the portion of the sensing signal comprises aninternally generated oscillator signal or a direct path measured soundsignal.
 4. The processor-readable medium of any one of claims 1 to 2wherein the instructions to derive the physiological movement signal areconfigured to multiply an oscillator signal with the portion of thesensed reflected signal.
 5. The processor-readable medium of any one ofclaims 1 to 4 wherein the derived physiological movement signalcomprises one or more of a respiratory motion, gross motion, or acardiac motion, of a user within the cabin vicinity.
 6. Theprocessor-readable medium of any one of claims 1 to 5 wherein theevaluation of the derived physiological movement signal comprisesdetermining any one or more of breathing rate, amplitude of breathing,relative amplitude of breathing, cardiac rate, cardiac amplitude andrelative cardiac amplitude.
 7. The processor-readable medium of any oneof claims 1 to 6 wherein the processor-executable instructions compriseinstructions to sense vehicle characteristics based on a signal from oneor more vehicle sensors and generate the output based on the sensedvehicle characteristics.
 8. The processor-readable medium of any one ofclaims 1 to 7 wherein the processor-executable instructions compriseinstructions to sense vehicle characteristics based on a signal from oneor more vehicle sensors, and to adjust at least a portion of theproduced sensing signal based on the sensed vehicle characteristics. 9.The processor-readable medium of any one of claims 7 to 8 wherein thesensed vehicle characteristics comprise any one or more of vehiclespeed, door opening state, window opening state, engine revolutions,vehicle location, seat occupancy, seatbelt fastening state, seatposition, steering wheel grip status, steering wheel angle, airconditioning system status, fan setting, brake setting, gas pedalsetting, cabin light, cabin noise, and/or cabin temperature.
 10. Theprocessor-readable medium of any one of claims 1 to 9 further comprisingprocessor-executable instructions to evaluate, via a microphone coupledto the electronic device, a sensed audible verbal communication; andwherein the generated output based on an evaluation of the derivedphysiological movement signal is further based on the sensed audibleverbal communication.
 11. The processor-readable medium of any one ofclaims 1 to 10 wherein at least a portion of the produced sensing signalis a sound signal in a substantially inaudible sound range.
 12. Theprocessor-readable medium of claim 11 wherein the sound signal is a lowfrequency ultrasonic acoustic signal.
 13. The processor-readable mediumof claim 12 wherein the sound signal is a dual tone frequency modulatedcontinuous wave signal.
 14. The processor-readable medium of claim 13wherein the dual tone frequency modulated continuous wave signalcomprises a first sawtooth frequency change at least partiallyoverlapped with a second sawtooth frequency change in a repeatedwaveform.
 15. The processor-readable medium of any one of claims 10 to14 wherein the electronic device comprises an audio entertainment systemthat comprises a plurality of speakers and wherein the electronic deviceis configured to derive different physiological movement signals, eachderived physiological movement signal being associated with a differentspeaker of the plurality of speakers.
 16. The processor-readable mediumof claim 15 wherein the instructions to control producing a sensingsignal produce sensing signals in different sensing frequency ranges foreach different speaker of the plurality of speakers.
 17. Theprocessor-readable medium of any one of claims 1 to 16 wherein theinstructions to control sensing the reflected signal from the cabinvicinity of the vehicle, control sensing of reflected signals by using aplurality of microphones.
 18. The processor-readable medium of any oneof claims 1 to 17 further comprising processor-executable instructionsto control the electronic device to generate, with a speaker, a soundpresentation to either discourage or promote sleep by the user in thecabin vicinity.
 19. The processor-readable medium of any one of claims 1to 18 wherein the sound presentation comprises a breathing entrainmentexercise.
 20. The processor-readable medium of any one of claims 1 to 19wherein the electronic device comprises a vehicle navigation system. 21.The processor-readable medium of claim 20 wherein processorexecutable-instructions of the electronic device are configured to,based on the output from the evaluation of the derived physiologicalmovement signal, set a parameter for a navigation route provided withthe vehicle navigation system.
 22. The processor-readable medium of anyone of claims 1 to 21 wherein the electronic device comprises asemi-autonomous or autonomous vehicle operations control system.
 23. Theprocessor-readable medium of claim 22 wherein processorexecutable-instructions of the electronic device are configured to,based on the output from the evaluation of the derived physiologicalmovement signal, control any one or more of: movement of the vehicle,adjustment of a light condition of the cabin vicinity, adjustment ofelectrochromic glass transparency, movement of a seat of the cabinvicinity, adjustment of a braking parameter, adjustment of anacceleration parameter, adjustment of a suspension setting, adjustmentof window coverage, adjustment of an acoustic barrier, immobilization ofthe vehicle, engagement of vehicle ventilation and/or engagement ofvehicle cabin cooling/heating system.
 24. The processor-readable mediumof any one of claims 1 to 23 wherein the evaluation of the portion ofthe derived physiological movement signal comprises detection of any oneor more of sleepiness, fatigue state, a sleep stage and a time in asleep stage and/or a calculation of sleep score.
 25. Theprocessor-readable medium of any one of any one of claims 1 to 24wherein the evaluation of the portion of the derived physiologicalmovement signal by the electronic device comprises detection of one ormore of respiratory health related parameters, sleep disorderedbreathing related parameters, and/or cardiac health related parameters.26. The processor-readable medium of any one of claims 1 to 25 whereinthe evaluation of the portion of the derived physiological movementsignal comprises detection of a gesture.
 27. The processor-readablemedium of any one of claims 1 to 26, wherein the produced sensing signalcomprises an ultra-wide band (UWB) sound sensing signal generated asaudible white noise.
 28. The processor-readable medium of any one ofclaims 1 to 27 further comprising processor-executable instructions togenerate, in a setup process, probing signals to map distances withinthe cabin vicinity.
 29. The processor-readable medium of any one ofclaims 1 to 28 further comprising processor-executable instructions todetect presence or absence of a user in the cabin vicinity based on theportion of the derived physiological movement signal.
 30. Theprocessor-readable medium of any one of claims 1 to 29 furthercomprising processor-executable instructions to conduct biometricrecognition of a user in the cabin vicinity based on the portion of thederived physiological movement signal.
 31. The processor-readable mediumof claim 30, wherein the output is based on the biometric recognitionand comprises at least one of: (a) generating an alert; and (b)controlling enabling and disabling a vehicle operations control systemof the vehicle.
 32. The processor-readable medium of any one of claims 1to 31 further comprising processor-executable instructions to filter asound signal sensed by a microphone coupled to the electronic device,the filter configured to mitigate vehicular sounds.
 33. Theprocessor-readable medium of claim 32 wherein the vehicular soundsinclude one or more of: motor noise, wind noise, a car horn, a doorclosing, and infotainment sounds.
 34. The processor-readable medium ofany one of claims 1 to 33 wherein the evaluation of the portion of thederived physiological movement signal by the electronic device comprisesclassification of the derived physiological movement signal, wherein theclassification evaluates features determined from the derivedphysiological movement signal by a deep belief network.
 35. Theprocessor-readable medium of any one of claims 1 to 34 wherein theevaluation of the portion of the derived physiological movement signalby the electronic device comprises a determination of a child remainingalone in the cabin vicinity, and wherein the output comprises: (a) agenerated warning, or (b) the vehicle operations control systeminitiating a ventilation and/or temperature condition of the cabinvicinity.
 36. The processor-readable medium of any one of claims 1 to 35further comprising processor-executable instructions to record databased on the derived physiological movement signal in a blockchain datasystem.
 37. The processor-readable medium of any one of claims 1 to 36further comprising processor-executable instructions to generate theoutput as an interactive language process through a chatbot program. 38.The processor-readable medium of any one of claims 1 to 37 wherein theelectronic device comprises a hand-held processing device.
 39. Theprocessor-readable medium of any one of claims 1 to 38 wherein theelectronic device comprises one or more integrated components of avehicle or a vehicular processing device.
 40. The processor-readablemedium of claim 39 wherein the electronic device comprises an audioentertainment system, wherein at least a portion of the sensing signalis combined with an audio entertainment content signal, and wherein thecombined sensing signal and audio entertainment content signal areproduced by one or more speakers of the audio entertainment system. 41.A server with access to the processor-readable medium of anyone ofclaims 1 to 40, wherein the server is configured to receive requests fordownloading the processor-executable instructions of theprocessor-readable medium to an electronic device or a vehicularprocessing device over a network.
 42. An electronic device comprising:one or more processors arranged to be coupled to a sensor operating in acabin vicinity of a vehicle; and (a) a processor-readable medium of anyone of claims 1 to 40 or (b) a processor-readable medium configured toaccess the processor-executable instructions of the server of claim 41.43. The electronic device of claim 42 wherein the sensor comprises atleast one of: (a) a speaker and microphone, (b) an infrared emitter anddetector, or (c) a radio frequency transceiver.
 44. The electronicdevice of any one of claims 42 to 43 wherein the electronic devicecomprises one or more integrated components of a vehicle or a vehicularprocessing device.
 45. The electronic device of claim 44 wherein theelectronic device comprises any one of more of a vehicle audioentertainment system, a vehicle navigation system, and a semi-autonomousor autonomous vehicle operations control system.
 46. The electronicdevice of any one of claims 42 to 45 further comprising at least oneportable component.
 47. The electronic device of claim 46, wherein theat least one portable component comprises a smart phone, a smart watchor smart jewelry.
 48. A method of a server having access to theprocessor-readable medium of any one of claims 1 to 40, or to theelectronic device of any one of claims 42 to 47, the method comprisingreceiving, at the server, a request for downloading theprocessor-executable instructions of the processor-readable medium tothe electronic device over a network; and transmitting theprocessor-executable instructions to the electronic device in responseto the request.
 49. A method of a processor of an electronic devicecomprising: accessing, with the processor, the processor-readable mediumof any one of claims 1 to 40, and executing, in the processor, theprocessor-executable instructions of the processor-readable medium. 50.A method of one or more processors of an electronic device to detectphysiological movement of a user in a cabin vicinity of a vehicle, themethod comprising: controlling producing a sensing signal in the cabinvicinity of the vehicle; controlling sensing, with a sensor, a reflectedsignal from the cabin vicinity of the vehicle; deriving a physiologicalmovement signal with at least a portion of the sensed reflected signaland a signal representative of at least a portion of the sensing signal;and generating an output based on an evaluation of at least a portion ofthe derived physiological movement signal.
 51. The method of claim 50wherein the sensing signal is anyone or more of a radio frequencysensing signal generated by a radio frequency transmitter coupled withthe electronic device, an acoustic sensing signal generated by a speakercoupled with the electronic device, and an infrared sensing signalgenerated by an infrared emitter coupled with the electronic device. 52.The method of any one of claims 50 to 51 wherein the signalrepresentative of the portion of the sensing signal comprises aninternally generated oscillator signal or a direct path measured signal.53. The method of any one of claims 50 to 52 wherein deriving thephysiological movement signal comprises multiplying an oscillator signalwith the portion of the sensed reflected signal.
 54. The method of anyone of claims 50 to 53 wherein the derived physiological movement signalcomprises one or more of a respiratory motion, a cardiac motion or grossmotion, of a user within the cabin vicinity.
 55. The method of any oneof claims 50 to 54 wherein the evaluation of the portion of the derivedphysiological movement signal comprises determining any one or more ofbreathing rate, relative amplitude of breathing, amplitude of breathing,cardiac rate, relative cardiac amplitude, and cardiac amplitude.
 56. Themethod of any one of claims 50 to 55 further comprising sensing vehiclecharacteristics based on a signal from one or more vehicle sensors andgenerating the output based on the sensed vehicle characteristics. 57.The method of any one of claims 50 to 56 further comprising sensingvehicle characteristics based on a signal from one or more vehiclesensors, and to adjust at least a portion of the produced sensing signalbased on the sensed vehicle characteristics.
 58. The method of any oneof claims 50 to 57 wherein the sensed vehicle characteristics compriseany one or more of vehicle speed, door opening state, window openingstate, engine revolutions, vehicle location, seat occupancy, seatbeltfastening state, seat position, steering wheel grip status, steeringwheel angle, air conditioning system status, fan setting, brake setting,gas pedal setting, cabin light, cabin noise, and/or cabin temperature.59. The method of any one of claims 50 to 58 further evaluating, via amicrophone coupled to the electronic device, a sensed audible verbalcommunication; and wherein the generated output based on an evaluationof the portion of the derived physiological movement signal is furtherbased on the sensed audible verbal communication.
 60. The method of anyone of claims 50 to 59 wherein at least a portion of the producedsensing signal is a sound signal in a substantially inaudible soundrange.
 61. The method of claim 60 wherein the sound signal is a lowfrequency ultrasonic acoustic signal.
 62. The method of claim 61 whereinthe sound signal is a dual tone frequency modulated continuous wavesignal.
 63. The method of claim 62 wherein the dual tone frequencymodulated continuous wave signal comprises a first sawtooth frequencychange at least partially overlapped with a second sawtooth frequencychange in a repeated waveform.
 64. The method of any one of claims 59 to63 wherein the electronic device comprises an audio entertainment systemcomprises a plurality of speakers and wherein the electronic devicederives different physiological movement signals, each derivedphysiological movement signal being associated with a different speakerof the plurality of speakers.
 65. The method of claim 64 wherein thecontrolling producing a sensing signal produces sensing signals indifferent sensing frequency ranges for each different speaker of theplurality of speakers.
 66. The method of any one of claims 50 to 65controlling sensing the reflected signal from the cabin vicinity of thevehicle comprises controlling sensing of reflected signals by using aplurality of microphones.
 67. The method of any one of claims 50 to 66further comprising controlling the electronic device to generate, with aspeaker, a sound presentation to either discourage or promote sleep bythe user in the cabin vicinity.
 68. The method of any one of claims 50to 67 wherein the sound presentation comprises a breathing entrainmentexercise.
 69. The method of any one of claims 50 to 68 wherein theelectronic device comprises a vehicle navigation system.
 70. The methodof claim 69 the electronic device, based on the output from theevaluation of the portion of the derived physiological movement signal,sets a parameter for a navigation route provided with the vehiclenavigation system.
 71. The method of any one of claims 50 to 70 whereinthe electronic device comprises a semi-autonomous or autonomous vehicleoperations control system.
 72. The method of claim 71 wherein theelectronic device controls, based on the output from the evaluation ofthe derived physiological movement signal, any one or more of: movementof the vehicle, adjustment of a light condition of the cabin vicinity,adjustment of electrochromic glass transparency, movement of a seat ofthe cabin vicinity, adjustment of a braking parameter, adjustment of anacceleration parameter, adjustment of a suspension setting, adjustmentof window coverage, adjustment of an acoustic barrier, immobilization ofthe vehicle, engagement of vehicle ventilation and/or engagement ofvehicle cabin cooling/heating system.
 73. The method of any one ofclaims 50 to 72 wherein the evaluation of the portion of the derivedphysiological movement signal comprises detecting any one or more ofsleepiness, fatigue state, a sleep stage and a time in a sleep stageand/or a calculation of sleep score.
 74. The method of any one of anyone of claims 50 to 73 wherein the evaluation of the portion of thederived physiological movement signal by the electronic device comprisesdetection of any one or more of respiratory health related parameters,sleep disordered breathing related parameters, and cardiac healthrelated parameters.
 75. The method of any one of claims 50 to 74 whereinthe evaluation of the portion of the derived physiological movementsignal detects a gesture.
 76. The method of any one of claims 50 to 75,wherein the produced sensing signal comprises an ultra-wide band (UWB)sound sensing signal generated as audible white noise.
 77. The method ofany one of claims 50 to 76 further generating, in a setup process,probing signals to map distances within the cabin vicinity.
 78. Themethod of any one of claims 50 to 77 further comprising detectingpresence and absence of a user in the cabin vicinity based on thederived physiological movement signal.
 79. The method of any one ofclaims 50 to 78 further comprising conducting biometric recognition of auser in the cabin vicinity based on the derived physiological movementsignal.
 80. The method of claim 79, wherein the output is based on thebiometric recognition and comprises at least one of: (a) generating analert and (b) controlling enabling and disabling a vehicle operationscontrol system of the vehicle.
 81. The method of any one of claims 50 to80 further comprising filtering a sound signal sensed by a microphonecoupled to the electronic device, the filtering configured to mitigatevehicular sounds.
 82. The method of claim 81 wherein the vehicularsounds include one or more of: motor noise, wind noise, a car horn, adoor closing, and infotainment sounds.
 83. The method of any one ofclaims 50 to 82 wherein the evaluation of the portion of the derivedphysiological movement signal by the electronic device comprisesclassifying the derived physiological movement signal, wherein theclassifying evaluates features determined from the derived physiologicalmovement signal by a deep belief network.
 84. The method of any one ofclaims 50 to 83 wherein the evaluation of the portion of the derivedphysiological movement signal by the electronic device comprisesdetermining a presence of a child remaining alone in the cabin vicinity,and wherein the output comprises: (a) a generated warning, or (b) thevehicle operations control system initiating a ventilation and/ortemperature condition of the cabin vicinity provided by a vehicleenvironment control system.
 85. The method of any one of claims 50 to 84further comprising recording data based on the derived physiologicalmovement signal in a blockchain data system.
 86. The method of any oneof claims 50 to 85 further comprising generating the output as aninteractive language process through a chatbot program.
 87. The methodof any one of claims 50 to 86 wherein the electronic device comprises ahand-held processing device.
 88. The method of any one of claims 50 to87 wherein the electronic device comprises one or more integratedcomponents of a vehicle or a vehicular processing device.
 89. The methodof claim 88 wherein the electronic device comprises an audioentertainment system, wherein at least a portion of the sensing signalis combined with an audio entertainment content signal, and wherein thecombined sensing signal and audio entertainment content signal areproduced by one or more speakers of the audio entertainment system.